Skip to content

Cuda performance significantly down if auto duration enabled in ACE-Step 1.5 task #389

Description

@Dev0Lab1

Enabling auto for duration (e.g., setting it -1) impacts performance significantly, almost 3 times slower task completions.
The following data is taken directly from the logs provided here.
Total of 8 runs, 4 for each cuda 0.7.0 and 0.7.1 releases (note however this issue might be present in the older releases too, not tested here).

duration=-1 duration=160 log 0.7.0
673 174 ace_step.planner.code.cfg_decode.graph.prepare_ms
4468 1246 ace_step.planner.code.prefill_cache_steps
94439 17808 ace_step.planner.code.decode.graph.step_ms
96601 19511 ace_step.planner.code.phase_ms
99422 22572 ace_step.planner.total_ms
99422 22572 ace_step.session.planner_generate_ms
84 24 ace_step.session.planner_release.graph.workspace_ms
68 47 ace_step.diffusion.cross_cache_ms
133127 49882 session.wall_ms
duration=-1 duration=160 log 0.7.1
700 183 ace_step.planner.code.cfg_decode.graph.prepare_ms
4468 1246 ace_step.planner.code.prefill_cache_steps
94442 17816 ace_step.planner.code.decode.graph.step_ms
96613 19516 ace_step.planner.code.phase_ms
99387 22547 ace_step.planner.total_ms
99387 22547 ace_step.session.planner_generate_ms
84 26 ace_step.session.planner_release.graph.workspace_ms
2 1 ace_step.condition_encoder.project_text_ms
69 44 ace_step.diffusion.cross_cache_ms
133060 49853 session.wall_ms
duration=-1 duration=160 log 0.7.0
704 182 ace_step.planner.code.cfg_decode.graph.prepare_ms
4468 1246 ace_step.planner.code.prefill_cache_steps
94431 17787 ace_step.planner.code.decode.graph.step_ms
96621 19507 ace_step.planner.code.phase_ms
99404 22396 ace_step.planner.total_ms
99404 22396 ace_step.session.planner_generate_ms
85 25 ace_step.session.planner_release.graph.workspace_ms
68 49 ace_step.diffusion.cross_cache_ms
133150 49715 session.wall_ms
duration=-1 duration=160 log 0.7.1
695 183 ace_step.planner.code.cfg_decode.graph.prepare_ms
4468 1246 ace_step.planner.code.prefill_cache_steps
94443 17818 ace_step.planner.code.decode.graph.step_ms
96611 19514 ace_step.planner.code.phase_ms
99379 22496 ace_step.planner.total_ms
99379 22496 ace_step.session.planner_generate_ms
84 25 ace_step.session.planner_release.graph.workspace_ms
2 1 ace_step.condition_encoder.project_text_ms
68 44 ace_step.diffusion.cross_cache_ms
133107 49806 session.wall_ms

That's consistently 133 vs 50 seconds. This means that the exact same task is taking (2.7) almost 3 times more to complete if duration set auto.
System: RTX 3060

  • depending on different hardware setup your milage may vary

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions