Skip to content

Commit fadd9f1

Browse files
authored
Merge pull request #15574 from nextcloud/docs/text2image-flux
Docs for text2image flux and image2image
2 parents 57e3149 + 90168ff commit fadd9f1

5 files changed

Lines changed: 136 additions & 1 deletion

File tree

‎admin_manual/ai/app_assistant.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,8 @@ Text-To-Image
8585

8686
In order to make use of Text-To-Image features, you will need an app that provides an image generation backend:
8787

88-
* :ref:`tex2image_stablediffusion2<ai-app-text2image_stablediffusion2>` (Customer support available upon request)
88+
* :ref:`text2image_stablediffusion2<ai-app-text2image_stablediffusion2>` (Customer support available upon request)
89+
* :ref:`text2image_flux<ai-app-text2image_flux>` - Local text-to-image and image editing with FLUX.2 [klein] 4B (Customer support available upon request)
8990
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
9091
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)
9192

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
===================================================
2+
App: Local Image Generation: Flux (text2image_flux)
3+
===================================================
4+
5+
.. _ai-app-text2image_flux:
6+
7+
The *text2image_flux* app is one of the apps that provide image generation functionality in Nextcloud and act as an
8+
image generation backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>` and :ref:`other apps making use of
9+
the image generation functionality<t2i-consumer-apps>`. It also provides image editing (image-to-image) from a text
10+
prompt. The *text2image_flux* app specifically runs only open source models and does so entirely on-premises.
11+
Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
12+
13+
Providers
14+
---------
15+
16+
The app registers the following Task Processing providers:
17+
18+
* **Nextcloud Local Image Generation: Flux 2 Klein 4B** - text-to-image (``core:text2image``)
19+
* **Nextcloud Local Image Generation: Flux 2 Klein 4B (Enhanced)** - text-to-image with automatic prompt refinement
20+
via a text-to-text provider (``core:text2text``) before generation
21+
* **Nextcloud Local Image Editing: Flux 2 Klein 4B** - image editing from a prompt (``core:image2image`` on
22+
Nextcloud 36+; a custom ``text2image_flux:image2image`` task type on Nextcloud 35 and older)
23+
24+
Select the providers in the Nextcloud AI admin settings. For the Enhanced provider, also install and enable a
25+
text-to-text backend such as :ref:`llm2<ai-app-llm2>`.
26+
27+
Default output size is ``1024x1024``. You can override the size per task (``WxH`` format) or change the default via the
28+
``DEFAULT_SIZE`` deploy option (see :ref:`Configuration <ai-app-text2image_flux-configuration>` below).
29+
30+
Requirements
31+
------------
32+
33+
* Minimal Nextcloud version: 30
34+
* This app is built as an External App and thus depends on AppAPI v3.1.0 or higher
35+
* Nextcloud AIO is supported
36+
* We currently support NVIDIA GPUs and x86_64 CPUs
37+
* CUDA >= v12.2 on your host system
38+
* GPU Sizing
39+
40+
* A NVIDIA GPU with at least 8GB VRAM (roughly 7GB used)
41+
* At least 8GB of system RAM
42+
43+
* CPU Sizing
44+
45+
* CPU inference is supported but very slow; for faster generation prefer a GPU or use
46+
:ref:`text2image_stablediffusion2<ai-app-text2image_stablediffusion2>`
47+
* At least 12GB of system RAM
48+
* The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores
49+
* The app will hog all cores by default, so it is usually better to run it on a separate machine
50+
51+
Installation
52+
------------
53+
54+
0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
55+
1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
56+
2. Install the **Local Image Generation: Flux** ExApp via the "Apps" page in the Nextcloud web admin user interface
57+
3. Optionally install a text-to-text provider such as :ref:`llm2<ai-app-llm2>` if you want to use the Enhanced
58+
text-to-image provider
59+
4. Select the Flux providers in the Nextcloud AI admin settings
60+
61+
.. _ai-app-text2image_flux-configuration:
62+
63+
Configuration
64+
-------------
65+
66+
You can set the following environment variables in the :ref:`Deploy Options <ai-app_api_deploy_options>` when installing
67+
or redeploying the ExApp:
68+
69+
* ``NUM_INFERENCE_STEPS`` - Number of denoising steps used when generating or editing images. Default is ``4``.
70+
* ``DEFAULT_SIZE`` - Default size of generated or edited images in ``WxH`` format (for example ``1024x1024``). Used when
71+
no size is provided with the task. Default is ``1024x1024``.
72+
73+
Scaling
74+
-------
75+
76+
It is currently not possible to scale this app, we are working on this. Throughput is lower than with
77+
:ref:`text2image_stablediffusion2<ai-app-text2image_stablediffusion2>` because Flux generation is slower. We do
78+
appreciate real-world feedback on capacity.
79+
80+
App store
81+
---------
82+
83+
You can also find the app in our app store, where you can write a review:
84+
`<https://apps.nextcloud.com/apps/text2image_flux>`_
85+
86+
Repository
87+
----------
88+
89+
You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features:
90+
`<https://github.com/nextcloud/text2image_flux>`_
91+
92+
Nextcloud customers should file bugs directly with our Support system.
93+
94+
Ethical AI Rating
95+
-----------------
96+
97+
**Rating: Yellow**
98+
99+
Positive:
100+
101+
* The software for training and inferencing of this model is open source
102+
* The trained model is freely available under Apache 2.0, and thus can be run on-premises
103+
104+
Negative:
105+
106+
* The training data is not freely available, making it not possible to check or correct for bias or optimise the
107+
performance and CO2 usage
108+
109+
Learn more about the Nextcloud Ethical AI Rating
110+
`in our blog <https://nextcloud.com/blog/nextcloud-ethical-ai-rating/>`_.
111+
112+
Known Limitations
113+
-----------------
114+
115+
* Image generation and editing are slower than with
116+
:ref:`text2image_stablediffusion2<ai-app-text2image_stablediffusion2>`; prefer that app if speed matters more than quality
117+
* The Enhanced provider depends on a working text-to-text Task Processing provider; if prompt improvement fails, the original prompt is used
118+
* Faces and people in general may not be generated properly
119+
* The results for certain image generation requests can be biased and may enforce stereotypes
120+
* We currently only support languages that the underlying model supports; correctness of language use in languages other
121+
than English may be poor depending on the language's coverage in the model's training data
122+
* Make sure to test the app for whether it meets the use-case's quality requirements
123+
* Customer support is available upon request, however we can't solve false or problematic output, most performance
124+
issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the
125+
implementation of the app (connectors, API, front-end, AppAPI)

‎admin_manual/ai/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Artificial Intelligence
1111
app_llm2
1212
app_stt_whisper2
1313
app_text2image_stablediffusion2
14+
app_text2image_flux
1415
app_recognize
1516
app_context_chat
1617
app_context_agent

‎admin_manual/ai/overview.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@ Nextcloud uses modularity to separate raw AI functionality from the Graphical Us
5454
"","`OpenAI and LocalAI integration (via MistralAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Orange","No","Yes","No","No"
5555
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - Whisper models by OpenAI","No","No"
5656
"Image generation","`Local Stable Diffusion 2 (ExApp) <https://apps.nextcloud.com/apps/text2image_stablediffusion2>`_","Yellow","Yes","Yes - StableDiffusion XL model by StabilityAI","No","Yes"
57+
"","`Local Image Generation: Flux (ExApp) <https://apps.nextcloud.com/apps/text2image_flux>`_","Yellow","Yes","Yes - FLUX.2 [klein] 4B model by Black Forest Labs","No","Yes"
5758
"","`Replicate integration <https://apps.nextcloud.com/apps/integration_replicate>`_","Yellow","Yes","Yes - StableDiffusion models by StabilityAI","No","No"
5859
"","`OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_","Red","No","No","No","No"
5960
"","`OpenAI and LocalAI integration (via LocalAI) <https://apps.nextcloud.com/apps/integration_openai>`_","Green","Yes","Yes","Yes","Yes"
@@ -206,6 +207,7 @@ Backend apps
206207
~~~~~~~~~~~~
207208

208209
* `Local Stable Diffusion 2 (ExApp) <https://apps.nextcloud.com/apps/text2image_stablediffusion2>`_ (Customer support available upon request)
210+
* :ref:`Local Image Generation: Flux (ExApp)<ai-app-text2image_flux>` (Customer support available upon request)
209211
* `OpenAI and LocalAI integration (via OpenAI API) <https://apps.nextcloud.com/apps/integration_openai>`_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
210212
* *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service<ai-ai_as_a_service>`)
211213

‎developer_manual/digging_deeper/task_processing.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,12 @@ The following built-in task types are available:
142142
* ``numberOfImages``: ``Number``
143143
* Output shape:
144144
* ``output``: ``ListOfImages``
145+
* ``'core:image2image'``: This task type is for editing images based on a text description of the changes. It is implemented by ``\OCP\TaskProcessing\TaskTypes\ImageToImage``
146+
* Input shape:
147+
* ``input``: ``ListOfImages``
148+
* ``prompt``: ``Text``
149+
* Output shape:
150+
* ``output``: ``Image``
145151
* ``'core:text2text:changetone'``: This task type is for reformulating a text, changing its tone. It is implemented by ``\OCP\TaskProcessing\TaskTypes\TextToTextChangeTone``
146152
* Input shape:
147153
* ``input``: ``Text``

0 commit comments

Comments
 (0)