How to make the conversational application created using dify use streaming output?

[majestichou]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Provide a description of the help you need

How to make the conversational application created using dify use streaming output? I deploy Dify by docker compose according to official docs Docker Compose Deployment
I create an application using Dify with Qwen API. But when I talk to the big language model, the AI's answer is not streaming output.
Does Dify have parameter configuration and can use streaming output?

[dosubot[bot]]

Hey there, @majestichou! Great to see you diving into another interesting challenge.

To enable streaming output for a conversational application when integrating with the Qwen API, you can set the "response_mode": "streaming" in your API request. This configuration allows for a continuous, stream-based interaction between the application and the user, which is suitable for applications requiring real-time responses and updates. Here's the relevant part of the request that configures streaming output:

"response_mode": "streaming",

This is demonstrated in the sample call for chat-messages in the Dify API documentation.

As for the invoke_llm method, there was a suggestion in a closed issue to modify it to support streaming. The conceptual example provided was:

llm_result: LLMResult = model_instance.invoke_llm(
    prompt_messages=prompt_messages,
    model_parameters=app_orchestration_config.model_config.parameters,
    tools=[],
    stop=app_orchestration_config.model_config.stop,
    stream=True,  # Enable streaming
    user=self.user_id,
    callbacks=[],
)

In this example, setting stream=True would enable streaming output. However, it's important to note that this would require modifications to the invoke_llm method and the underlying model to support streaming. Also, keep in mind that streaming output can increase the complexity of your application and may require additional resources, such as more memory or CPU. Therefore, it's important to carefully consider the trade-offs before implementing this feature.

Unfortunately, I wasn't able to find specific information on whether the invoke_llm method currently supports the stream parameter or what changes would be required in the Dify codebase to support it. It's possible that this information is available elsewhere or I may have missed it. If you have more specific information or context that could help me find a more accurate answer, please provide it.

Sources

• Agent mode does not use streaming output, so the conversation experience seems slow
• Streaming mode not working as expected
• en/user-guide/launching-dify-apps/developing-with-apis/README.md

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.