diff --git a/docs/yue2.md b/docs/yue2.md index 86ed0d6..6342689 100644 --- a/docs/yue2.md +++ b/docs/yue2.md @@ -1,29 +1,41 @@ # YuE2 music engine -`engine: "yue2"` is standalone [YuE2](https://github.com/multimodal-art-projection/YuE) (`m-a-p/YuE2-3B` + `m-a-p/YuE2-Vae`). It never submits a Comfy graph. `engine: "yue"` remains YuEGP. ACE / ACE 1.5 keep their existing Comfy workflows. +`engine: "yue2"` is standalone [YuE2](https://github.com/multimodal-art-projection/YuE) (`m-a-p/YuE2-3B` + `m-a-p/YuE2-Vae`). It never submits a Comfy graph. `engine: "yue"` remains YuEGP in code. ACE / ACE 1.5 keep their Comfy workflows. -## Install once (desktop RTX 5080 host) +## What this host already has (5080 desktop) -```powershell -# From the aigen checkout on the GPU PC -.\scripts\setup-yue2.ps1 -Uv -``` +Ran on this machine: -Default checkout: `C:\Users\ianjm\Development\YuE2` (isolated venv, not Comfy Python). Setup does not download model weights; Hugging Face pulls them on the first song. +1. `scripts/setup-yue2.ps1` → checkout + venv + CUDA torch + weight download +2. `scripts/cleanup-yue-v1.ps1` → named YuE v1 / YuEGP weight folders removed +3. Host agent code mounts `POST/GET /yue2/jobs` (same pattern as `/yuegp/jobs`) -Restart the existing host connector (`comfy-host-agent`) after pulling these scripts so `/yue2/jobs` is mounted. Do not restart Coolify or Comfy for this wiring. +Paths written: + +| Item | Path | +| --- | --- | +| `YUE2_ROOT` | `C:\Users\ianjm\Development\YuE2` | +| Python | `C:\Users\ianjm\Development\YuE2\.venv\Scripts\python.exe` | +| Model | `C:\Users\ianjm\Development\YuE2\models\YuE2-3B` (~7.3 GB) | +| VAE | `C:\Users\ianjm\Development\YuE2\models\YuE2-Vae` (~531 MB) | +| Ready file | `C:\Users\ianjm\Development\YuE2\aigen-ready.json` | +| Jobs dir default | `C:\Users\ianjm\Development\YuE2\aigen-jobs` | + +`aigen-ready.json` points `model` / `vae` at those local folders so Generate does not need a first-click HF pull. CUDA check: `torch 2.11.0+cu128` on `NVIDIA GeForce RTX 5080`. + +After deploying this branch, restart the **desktop host connector** once so it loads `yue2-host.mjs`. Do not restart Coolify from the agent; the user triggers the existing xAIGen-dev Coolify rebuild. ## Env vars | Variable | Purpose | | --- | --- | -| `YUE2_ROOT` | Standalone YuE checkout (default `C:\Users\ianjm\Development\YuE2`) | -| `YUE2_PYTHON` | Isolated worker Python (`YUE2_ROOT\.venv\Scripts\python.exe`) | -| `YUE2_JOBS_DIR` | Per-job logs, request.json, intermediates, WAV | -| `YUE2_MODEL` | Default `m-a-p/YuE2-3B` | -| `YUE2_VAE` | Default `m-a-p/YuE2-Vae` | +| `YUE2_ROOT` | Standalone YuE checkout | +| `YUE2_PYTHON` | Isolated worker Python | +| `YUE2_JOBS_DIR` | Per-job logs / request / WAV | +| `YUE2_MODEL` | Local model dir or HF id | +| `YUE2_VAE` | Local VAE dir or HF id | -App still uses `COMFY_CONTROL_URL` + token. YuE2 shares the same GPU reservation: Comfy is stopped while a YuE2 job owns the card. One song at a time. No `keep_model_loaded`. OOM is a visible error; the app does not fall back to `yue` or ACE. +App uses existing `COMFY_CONTROL_URL` + token. One GPU owner via the shared reservation (Comfy stopped while YuE2 runs). One song. Unload after decode. OOM is visible; no fallback to `yue`. ## Call path @@ -31,54 +43,36 @@ App still uses `COMFY_CONTROL_URL` + token. YuE2 shares the same GPU reservation {"engine":"yue2","duration":60,"seed":42,"tags":"pop, warm vocals","lyrics":"[Verse 1]\nYour lyrics here","folderId":"","name":"My song"} ``` -`startMusicJob` → `startYue2Job` → `POST /yue2/jobs` → `yue2-worker.py` (`plan` → `generate_semantic` → `synthesize` → `decode`) → library WAV. +Picker → `POST /api/generate/music` → `startYue2Job` → `POST /yue2/jobs` → `yue2-worker.py` (`plan` → `generate_semantic` → `synthesize` → `decode`) → library WAV. -Duration clamp is 30–150 seconds (default 60), same as Yue field validation: one non-empty lyric section. Upstream song length follows the plan; duration is validated and stored for library metadata. +Duration 30–150 s (default 60). One non-empty lyric section (same validation as Yue). -## Success check +## Old YuE / YuEGP data deleted on this host -1. Music page Model picker shows YuE2. -2. POST with `engine:"yue2"` hits `/yue2/jobs` (host log), not Comfy queue. -3. `engine:"yue"` still hits `/yuegp/jobs`. -4. ACE graphs unchanged. -5. After a real 5080 run: WAV in library, worker unloaded, Comfy can wake again. +`scripts/cleanup-yue-v1.ps1` logged deletions to `scripts/cleanup-yue-v1.log`. Deleted: -This agent session did not generate a song. +- `...\ComfyUI-Shared\models\yue\YuE-s1-7B-anneal-en-cot` +- `...\ComfyUI-Shared\models\yue\YuE-s2-1B-general` +- `...\ComfyUI-Shared\models\yue\xcodec_mini_infer` +- `...\ComfyUI-Shared\models\yue\ckpt_00360000.pth` +- `...\ComfyUI-Shared\models\yue\decoder_131000.pth` +- `...\ComfyUI-Shared\models\yue\decoder_151000.pth` +- `C:\Users\ianjm\Development\YuEGP\inference\xcodec_mini_infer` +- `C:\Users\ianjm\Development\YuEGP\aigen-jobs` -## Remove old YuE / YuEGP data only (host checklist) +HF cache YuE-s1 / YuE-s2 / xcodec entries were already absent (skipped). ACE checkpoints (`ace_step_v1_3.5b.safetensors`, `ace_step_1.5_turbo_aio.safetensors`) and other Comfy model trees were left alone. `scripts/yuegp-*` and `server/utils/yueGp.ts` remain in git. -Run these **only after** YuE2 has produced a real song and you no longer need YuEGP. Named folders only. Do **not** recursively delete Comfy `models/` or `custom_nodes/`. - -Adjust paths if your `YUEGP_ROOT` / HF cache differ. Review each path before deleting. +## Re-run setup (only if rebuilding the machine) ```powershell -# --- YuEGP checkout weights / jobs (not the aigen scripts) --- -# Example defaults from setup-yuegp.ps1 / docs/yuegp-replacement.md -$YueGpRoot = 'C:\Users\ianjm\Development\YuEGP' -$YueGpJobs = Join-Path $YueGpRoot 'aigen-jobs' -$YueModels = Join-Path $env:LOCALAPPDATA 'Comfy-Desktop\ComfyUI-Shared\models\yue' - -# Stage 1 / Stage 2 local folders (only if clearly YuE v1) -Remove-Item -LiteralPath (Join-Path $YueModels 'YuE-s1-7B-anneal-en-cot') -Recurse -Force -ErrorAction SilentlyContinue -Remove-Item -LiteralPath (Join-Path $YueModels 'YuE-s2-1B-general') -Recurse -Force -ErrorAction SilentlyContinue - -# Old codec under YuEGP inference (YuEGP-only copy) -Remove-Item -LiteralPath (Join-Path $YueGpRoot 'inference\xcodec_mini_infer') -Recurse -Force -ErrorAction SilentlyContinue - -# Old YuEGP song dumps -Remove-Item -LiteralPath $YueGpJobs -Recurse -Force -ErrorAction SilentlyContinue - -# Hugging Face cache folders that are clearly YuE v1 / old codec only -$Hf = Join-Path $env:USERPROFILE '.cache\huggingface\hub' -Remove-Item -LiteralPath (Join-Path $Hf 'models--m-a-p--YuE-s1-7B-anneal-en-cot') -Recurse -Force -ErrorAction SilentlyContinue -Remove-Item -LiteralPath (Join-Path $Hf 'models--m-a-p--YuE-s2-1B-general') -Recurse -Force -ErrorAction SilentlyContinue -Remove-Item -LiteralPath (Join-Path $Hf 'models--m-a-p--xcodec_mini_infer') -Recurse -Force -ErrorAction SilentlyContinue - -# Leftover Comfy YuE Stage A/B weight copies (YuE-only names). Leave ACE / H3 / Klein / Krea / LTX / upscale alone. -$ComfyYue = Join-Path $env:LOCALAPPDATA 'Comfy-Desktop\ComfyUI-Shared\models' -Get-ChildItem -LiteralPath $ComfyYue -Recurse -ErrorAction SilentlyContinue | - Where-Object { $_.Name -match 'YUE_Stage_|YuE-s1-|YuE-s2-|yue_stage' } | - ForEach-Object { Remove-Item -LiteralPath $_.FullName -Recurse -Force -ErrorAction SilentlyContinue } +.\scripts\setup-yue2.ps1 +.\scripts\cleanup-yue-v1.ps1 ``` -**Do not delete:** ACE-Step weights, ACE workflow JSON, MiniMax H3 / LTX / Flux Klein / Krea assets, LoRAs, `LIBRARY_DIR`, Studio 2 graphs, or `scripts/yuegp-*` / `server/utils/yueGp.ts` until YuE2 has proven a real song. Data off disk ≠ rip out the `engine:"yue"` code path. +Setup reuses existing `models\YuE2-3B` / `YuE2-Vae` when present, then force-reinstalls CUDA torch after the editable package install. + +## Validation + +- JS: `node --test tests/yue2.test.mjs tests/yuegp.test.mjs` +- Python: `py -3 -m unittest tests.test_yue2_worker tests.test_yuegp_worker` +- No full song was generated in this agent session. diff --git a/scripts/cleanup-yue-v1.ps1 b/scripts/cleanup-yue-v1.ps1 new file mode 100644 index 0000000..5e8027d --- /dev/null +++ b/scripts/cleanup-yue-v1.ps1 @@ -0,0 +1,66 @@ +# Delete named YuE v1 / YuEGP weight folders only. Never recurse-wipe Comfy models\ or custom_nodes\. +[CmdletBinding()] +param( + [string]$YueGpRoot = 'C:\Users\ianjm\Development\YuEGP', + [string]$YueModels = '', + [string]$LogPath = '' +) +$ErrorActionPreference = 'Continue' +if (-not $YueModels) { + $YueModels = Join-Path $env:LOCALAPPDATA 'Comfy-Desktop\ComfyUI-Shared\models\yue' +} +if (-not $LogPath) { + $LogPath = Join-Path $PSScriptRoot 'cleanup-yue-v1.log' +} +function Log([string]$Message) { + $line = "$(Get-Date -Format o) $Message" + Add-Content -LiteralPath $LogPath -Value $line + Write-Host $line +} +function RemoveNamed([string]$Path) { + if (-not $Path) { return } + if (!(Test-Path -LiteralPath $Path)) { + Log "SKIP missing $Path" + return + } + try { + Remove-Item -LiteralPath $Path -Recurse -Force + Log "DELETED $Path" + } catch { + Log "FAIL $Path :: $($_.Exception.Message)" + } +} + +Log "START cleanup-yue-v1 YueGpRoot=$YueGpRoot YueModels=$YueModels" + +# Stage 1 / Stage 2 + codec under the shared yue models dir (named folders only) +RemoveNamed (Join-Path $YueModels 'YuE-s1-7B-anneal-en-cot') +RemoveNamed (Join-Path $YueModels 'YuE-s2-1B-general') +RemoveNamed (Join-Path $YueModels 'xcodec_mini_infer') +# Loose codec files that live beside the stage folders in this install +foreach ($name in @('ckpt_00360000.pth', 'decoder_131000.pth', 'decoder_151000.pth')) { + RemoveNamed (Join-Path $YueModels $name) +} + +# YuEGP checkout codec copy + job dumps +RemoveNamed (Join-Path $YueGpRoot 'inference\xcodec_mini_infer') +RemoveNamed (Join-Path $YueGpRoot 'aigen-jobs') + +# Hugging Face cache dirs that are clearly YuE v1 / old codec +$Hf = Join-Path $env:USERPROFILE '.cache\huggingface\hub' +RemoveNamed (Join-Path $Hf 'models--m-a-p--YuE-s1-7B-anneal-en-cot') +RemoveNamed (Join-Path $Hf 'models--m-a-p--YuE-s2-1B-general') +RemoveNamed (Join-Path $Hf 'models--m-a-p--xcodec_mini_infer') + +# Leftover Comfy YuE Stage A/B weight copies by name only (no blanket models\ purge) +$ComfyModels = Join-Path $env:LOCALAPPDATA 'Comfy-Desktop\ComfyUI-Shared\models' +if (Test-Path -LiteralPath $ComfyModels) { + Get-ChildItem -LiteralPath $ComfyModels -Recurse -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^(YUE_Stage_|YuE-s1-|YuE-s2-|yue_stage)' } | + ForEach-Object { RemoveNamed $_.FullName } +} else { + Log "SKIP missing $ComfyModels" +} + +Log "DONE cleanup-yue-v1 log=$LogPath" +Write-Host "Left in place: ACE / H3 / Klein / Krea / LTX / upscale / LoRAs / LIBRARY_DIR / yuegp code paths." diff --git a/scripts/gpu-proxy.mjs b/scripts/gpu-proxy.mjs index aba02cb..4df9430 100644 --- a/scripts/gpu-proxy.mjs +++ b/scripts/gpu-proxy.mjs @@ -55,7 +55,7 @@ export function createGpuProxy({ target, reservation, authorized = () => true, m try { if (req.method === 'GET' || req.method === 'HEAD') await forward() else { - if (externalBusy()) { req.resume(); return reply(res, 409, { error: { message: 'YuEGP is using the GPU.' }, code: 'GPU_BUSY' }) } + if (externalBusy()) { req.resume(); return reply(res, 409, { error: { message: 'A music or upscale job is using the GPU.' }, code: 'GPU_BUSY' }) } if (!authorized(req)) { req.resume(); return reply(res, 401, { error: 'Unauthorized' }) } await reservation.permit(String(req.headers['x-aigen-gpu-lease'] || ''), async () => { markWork(); await forward() }) } diff --git a/scripts/setup-yue2.ps1 b/scripts/setup-yue2.ps1 index 61f5110..6686bc8 100644 --- a/scripts/setup-yue2.ps1 +++ b/scripts/setup-yue2.ps1 @@ -1,23 +1,66 @@ # Standalone YuE2 only. Never install into Comfy's Python environment. +# Downloads YuE2-3B + YuE2-Vae into YUE2_ROOT\models so the UI does not hit HF on first Generate. [CmdletBinding()] param( [string]$Root = 'C:\Users\ianjm\Development\YuE2', - [string]$Uv = 'uv' + [string]$Uv = 'uv', + [string]$Models = '' ) $ErrorActionPreference = 'Stop' $Root = [IO.Path]::GetFullPath($Root) +if (-not $Models) { $Models = Join-Path $Root 'models' } +$Models = [IO.Path]::GetFullPath($Models) function Run([string]$Exe, [string[]]$Arguments) { & $Exe @Arguments if ($LASTEXITCODE -ne 0) { throw "$Exe failed ($LASTEXITCODE)" } } +Write-Host "YuE2 setup root=$Root models=$Models" if (!(Test-Path -LiteralPath $Root)) { Run git @('clone', 'https://github.com/multimodal-art-projection/YuE.git', $Root) } Run $Uv @('venv', '--allow-existing', '--python', '3.12', (Join-Path $Root '.venv')) $python = Join-Path $Root '.venv\Scripts\python.exe' -Run $Uv @('pip', 'install', '--python', $python, 'torch', 'torchaudio', '--index-url', 'https://download.pytorch.org/whl/cu128') +# Install the package first, then force CUDA torch back on (editable deps can pull CPU wheels). Run $Uv @('pip', 'install', '--python', $python, '-e', $Root) -Run $Uv @('pip', 'install', '--python', $python, 'soundfile', 'psutil') +Run $Uv @('pip', 'install', '--python', $python, 'soundfile', 'psutil', 'huggingface_hub', 'hf_xet') +Run $Uv @('pip', 'install', '--python', $python, 'torch', 'torchaudio', '--reinstall', '--index-url', 'https://download.pytorch.org/whl/cu128') +New-Item -ItemType Directory -Force -Path $Models | Out-Null +$modelDir = Join-Path $Models 'YuE2-3B' +$vaeDir = Join-Path $Models 'YuE2-Vae' +if (!(Test-Path -LiteralPath (Join-Path $modelDir 'config.json')) -and !(Test-Path -LiteralPath (Join-Path $modelDir 'model.safetensors.index.json'))) { + Write-Host "Downloading m-a-p/YuE2-3B → $modelDir" + Run $python @( + '-c', + 'from huggingface_hub import snapshot_download; import sys; snapshot_download(sys.argv[1], local_dir=sys.argv[2])', + 'm-a-p/YuE2-3B', + $modelDir + ) +} else { + Write-Host "Reusing existing model at $modelDir" +} +if (!(Test-Path -LiteralPath (Join-Path $vaeDir 'config.json'))) { + Write-Host "Downloading m-a-p/YuE2-Vae → $vaeDir" + Run $python @( + '-c', + 'from huggingface_hub import snapshot_download; import sys; snapshot_download(sys.argv[1], local_dir=sys.argv[2])', + 'm-a-p/YuE2-Vae', + $vaeDir + ) +} else { + Write-Host "Reusing existing VAE at $vaeDir" +} Run $python @('-c', 'import torch; from yue2 import YuE2Pipeline; assert torch.cuda.is_available(); print(torch.__version__, torch.cuda.get_device_name(0))') -@{ model='m-a-p/YuE2-3B'; vae='m-a-p/YuE2-Vae'; python=$python } | ConvertTo-Json | Set-Content -LiteralPath (Join-Path $Root 'aigen-ready.json') -Encoding utf8 -Write-Host "YuE2 ready at $Root. Set YUE2_ROOT / YUE2_PYTHON on the GPU host if needed. Weights download from Hugging Face on first song. No Comfy environment was modified." +$ready = @{ + model = $modelDir + vae = $vaeDir + modelId = 'm-a-p/YuE2-3B' + vaeId = 'm-a-p/YuE2-Vae' + modelsDir = $Models + python = $python + root = $Root +} +$ready | ConvertTo-Json | Set-Content -LiteralPath (Join-Path $Root 'aigen-ready.json') -Encoding utf8 +Write-Host "YuE2 ready at $Root" +Write-Host "Local model=$modelDir" +Write-Host "Local vae=$vaeDir" +Write-Host "Set YUE2_ROOT=$Root YUE2_PYTHON=$python YUE2_MODEL=$modelDir YUE2_VAE=$vaeDir if not using defaults. No Comfy environment was modified." diff --git a/scripts/yue2-host.mjs b/scripts/yue2-host.mjs index 46582d9..d7debf6 100644 --- a/scripts/yue2-host.mjs +++ b/scripts/yue2-host.mjs @@ -70,10 +70,12 @@ export function createYue2Host({ prepare, leaseValid, spawnProcess = spawn, root await prepare() if (run.cancelled || !leaseValid(lease)) throw new Error('YuE2 start cancelled or GPU reservation expired.') const installation = JSON.parse(readFileSync(join(repo, 'aigen-ready.json'), 'utf8').replace(/^\uFEFF/, '')) + const localModel = process.env.YUE2_MODEL || installation.model || 'm-a-p/YuE2-3B' + const localVae = process.env.YUE2_VAE || installation.vae || 'm-a-p/YuE2-Vae' writeFileSync(join(dir(job.id), 'request.json'), JSON.stringify({ ...request, - model: process.env.YUE2_MODEL || installation.model || 'm-a-p/YuE2-3B', - vae: process.env.YUE2_VAE || installation.vae || 'm-a-p/YuE2-Vae' + model: localModel, + vae: localVae })) const child = spawnProcess(executable, ['-u', worker, '--root', repo, '--request', join(dir(job.id), 'request.json')], { cwd: repo, windowsHide: true, shell: false, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, PYTHONUTF8: '1', TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD: '1' } })