Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,32 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+samples/e1.png filter=lfs diff=lfs merge=lfs -text
+samples/e2.png filter=lfs diff=lfs merge=lfs -text
+samples/e3.png filter=lfs diff=lfs merge=lfs -text
+samples/e4.png filter=lfs diff=lfs merge=lfs -text
+samples/e5.png filter=lfs diff=lfs merge=lfs -text
+samples/e6.png filter=lfs diff=lfs merge=lfs -text
+samples/e7_1.png filter=lfs diff=lfs merge=lfs -text
+samples/e7_2.png filter=lfs diff=lfs merge=lfs -text
+samples/e7_3.png filter=lfs diff=lfs merge=lfs -text
+samples/e7_4.png filter=lfs diff=lfs merge=lfs -text
+samples/ic_target.png filter=lfs diff=lfs merge=lfs -text
+samples/inpaint_i1.jpg filter=lfs diff=lfs merge=lfs -text
+samples/inpaint_i2.png filter=lfs diff=lfs merge=lfs -text
+samples/inpaint_o1.png filter=lfs diff=lfs merge=lfs -text
+samples/inpaint_o2.png filter=lfs diff=lfs merge=lfs -text
+samples/ip_1.png filter=lfs diff=lfs merge=lfs -text
+samples/ip_2.png filter=lfs diff=lfs merge=lfs -text
+samples/ip_3.png filter=lfs diff=lfs merge=lfs -text
+samples/ip_ref.png filter=lfs diff=lfs merge=lfs -text
+samples/regional_attention.jpg filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_1_mask_0.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_2_mask_0.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_3_mask_27.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_4_mask_21.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_5_mask_0.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_6_mask_8.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_7_mask_5.png filter=lfs diff=lfs merge=lfs -text
+samples/styled_entity_control_example_7_mask_6.png filter=lfs diff=lfs merge=lfs -text
+samples/video.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,113 @@
+---
+license: apache-2.0
+---
+# EliGen: Entity-Level Controlled Image Generation
+
+## Introduction
+
+We propose EliGen, a novel approach that leverages fine-grained entity-level information to enable precise and controllable text-to-image generation. EliGen excels in tasks such as entity-level controlled image generation and image inpainting, while its applicability is not limited to these areas. Additionally, it can be seamlessly integrated with existing community models, such as the IP-Adapter and In-Context LoRA.
+
+* Paper: [EliGen: Entity-Level Controlled Image Generation with Regional Attention](https://arxiv.org/abs/2501.01097)
+* Github: [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio)
+* Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen)
+* Online Demo: [ModelScope EliGen Studio](https://www.modelscope.cn/studios/DiffSynth-Studio/EliGen)
+* Training dataset: [ModelScope Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet)
+
+## Methodology
+
+![regional-attention](./samples/regional_attention.jpg)
+
+We introduce a regional attention mechanism within the DiT framework to effectively process the conditions of each entity. This mechanism enables the local prompt associated with each entity to semantically influence specific regions through regional attention. To further enhance the layout control capabilities of EliGen, we meticulously contribute an entity-annotated dataset and fine-tune the model using the LoRA framework.
+
+1. **Regional Attention**: Regional attention is shown in the above figure, which can be easily applied to other text-to-image models. Its core principle involves transforming the positional information of each entity into an attention mask, ensuring that the mechanism only affects the designated regions.
+
+2. **Dataset with Entity Annotation**: To construct a dedicated entity control dataset, we start by randomly selecting captions from DiffusionDB and generating the corresponding source image using Flux. Next, we employ Qwen2-VL 72B, recognized for its advanced grounding capabilities among MLLMs, to randomly identify entities within the image. These entities are annotated with local prompts and bounding boxes for precise localization, forming the foundation of our dataset for further training.
+
+3. **Training**: We utilize LoRA (Low-Rank Adaptation) and DeepSpeed to fine-tune regional attention mechanisms using a curated dataset, enabling our EliGen model to achieve effective entity-level control.
+
+## Usage
+This model was trained using [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio). We recommend using DiffSynth-Studio for generation.
+```shell
+git clone https://github.com/modelscope/DiffSynth-Studio.git
+cd DiffSynth-Studio
+pip install -e .
+```
+1. **Entity-Level Controlled Image Generation**  
+   EliGen achieves effective entity-level control results. See [entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control.py) for usage.
+2. **Image Inpainting**  
+   To apply EliGen to image inpainting tasks, we propose an inpainting fusion pipeline that preserves non-inpainted areas while enabling precise, entity-level modifications within inpainted regions.  
+   See [entity_inpaint.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_inpaint.py) for usage.
+3. **Styled Entity Control**  
+   EliGen can be seamlessly integrated with existing community models. We provide an example of integrating it with IP-Adapter. See [entity_control_ipadapter.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control_ipadapter.py) for usage.
+4. **Entity Transfer**  
+   We provide an example of integrating EliGen with In-Context LoRA, achieving interesting entity transfer results. See [entity_transfer.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_transfer.py) for usage.
+5. **Play with EliGen using UI**  
+   Download the EliGen checkpoint from [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen) to `models/lora/entity_control` and run the following command to launch the interactive UI:  
+   ```bash
+   python apps/gradio/entity_level_control.py
+   ```
+
+## Examples
+### Entity-Level Controlled Image Generation
+
+1. Generating images with continuously changing entity positions.
+
+<center>	
+    <video muted="true" autoplay="true" loop="true" height="512" width="512" src="./samples/video.mp4"></video>
+</center>
+
+1. Image generation results for complex combinations of entities, demonstrating the strong generalization capability of EliGen. See [entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control.py) `example_1-6` for the generation prompts.
+
+|Entity Conditions|Generated Image|
+|-|-|
+|![eligen_example_1_mask_0](./samples/e1_m.png)|![eligen_example_1_0](./samples/e1.png)|
+|![eligen_example_2_mask_0](./samples/e2_m.png)|![eligen_example_2_0](./samples/e2.png)|
+|![eligen_example_3_mask_27](./samples/e3_m.png)|![eligen_example_3_27](./samples/e3.png)|
+|![eligen_example_4_mask_21](./samples/e4_m.png)|![eligen_example_4_21](./samples/e4.png)|
+|![eligen_example_5_mask_0](./samples/e5_m.png)|![eligen_example_5_0](./samples/e5.png)|
+|![eligen_example_6_mask_8](./samples/e6_m.png)|![eligen_example_6_8](./samples/e6.png)|
+
+2. Demonstration of the robustness of EliGen. The following examples are generated using the same prompt but different random seeds. Refer to [entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control.py) `example_7` for the generation prompt.
+
+|Entity Conditions|Generated Image|
+|-|-|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_5](./samples/e7_1.png)|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_6](./samples/e7_2.png)|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_7](./samples/e7_3.png)|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_8](./samples/e7_4.png)|
+
+### Image Inpainting
+Demonstration of the inpainting mode in EliGen. See [entity_inpaint.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_inpaint.py) for generation prompts.
+|Inpainting Input|Inpainting Output|
+|-|-|
+|![inpaint_i1](./samples/inpaint_i1.jpg)|![inpaint_o1](./samples/inpaint_o1.png)|
+|![inpaint_i2](./samples/inpaint_i2.png)|![inpaint_o2](./samples/inpaint_o2.png)|
+
+### Styled Entity Control
+Demonstration of styled entity control results using EliGen and IP-Adapter. See [entity_control_ipadapter.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control_ipadapter.py) for generation prompts.
+|Style Reference|Entity Control Variation 1|Entity Control Variation 2|Entity Control Variation 3|
+|-|-|-|-|
+|![ip_ref](./samples/ip_ref.png)|![ip_1](./samples/ip_1.png)|![ip_2](./samples/ip_2.png)|![ip_3](./samples/ip_3.png)|
+
+We also provide a demo of styled entity control results using EliGen with a specific style LoRA. See [./styled_entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/styled_entity_control.py) for details. Below is the visualization of EliGen combined with the [Lego DreamBooth LoRA](https://huggingface.co/merve/flux-lego-lora-dreambooth).
+|![image_1_base](./samples/styled_entity_control_example_1_mask_0.png)|![result1](./samples/styled_entity_control_example_2_mask_0.png)|![result2](./samples/styled_entity_control_example_3_mask_27.png)|![result3](./samples/styled_entity_control_example_4_mask_21.png)|
+|-|-|-|-|
+|![image_1_base](./samples/styled_entity_control_example_5_mask_0.png)|![result1](./samples/styled_entity_control_example_6_mask_8.png)|![result2](./samples/styled_entity_control_example_7_mask_5.png)|![result3](./samples/styled_entity_control_example_7_mask_6.png)|
+
+### Entity Transfer
+Demonstration of the entity transfer results using EliGen and In-Context LoRA. See [entity_transfer.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_transfer.py) for generation prompts.
+
+|Entity to Transfer|Transfer Target Image|Transfer Example 1|Transfer Example 2|
+|-|-|-|-|
+|![ic_logo](./samples/ic_logo.jpg)|![ic_target](./samples/ic_target.png)|![ic_1](./samples/ic_1.jpg)|![ic_2](./samples/ic_2.jpg)|
+
+## Citation
+If you find our work helpful, please consider citing us:
+```
+@article{zhang2025eligen,
+  title={Eligen: Entity-level controlled image generation with regional attention},
+  author={Zhang, Hong and Duan, Zhongjie and Wang, Xingjun and Chen, Yingda and Zhang, Yu},
+  journal={arXiv preprint arXiv:2501.01097},
+  year={2025}
+}
+```

README_from_modelscope.md

@@ -0,0 +1,143 @@
+---
+frameworks:
+- Pytorch
+license: Apache License 2.0
+tasks:
+- text-to-image-synthesis
+
+#model-type:
+##如 gpt、phi、llama、chatglm、baichuan 等
+#- gpt
+
+#domain:
+##如 nlp、cv、audio、multi-modal
+#- nlp
+
+#language:
+##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
+#- cn
+
+#metrics:
+##如 CIDEr、Blue、ROUGE 等
+#- CIDEr
+
+#tags:
+##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
+#- pretrained
+
+#tools:
+##如 vllm、fastchat、llamacpp、AdaSeq 等
+#- vllm
+base_model:
+  - black-forest-labs/FLUX.1-dev
+tags:
+  - lora
+---
+# EliGen: Entity-Level Controlled Image Generation
+
+## Introduction
+
+We propose EliGen, a novel approach that leverages fine-grained entity-level information to enable precise and controllable text-to-image generation. EliGen excels in tasks such as entity-level controlled image generation and image inpainting, while its applicability is not limited to these areas. Additionally, it can be seamlessly integrated with existing community models, such as the IP-Adpater and In-Cotext LoRA.
+
+* Paper: [EliGen: Entity-Level Controlled Image Generation with Regional Attention](https://arxiv.org/abs/2501.01097)
+* Github: [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio)
+* Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen)
+* Online Demo: [ModelScope EliGen Studio](https://www.modelscope.cn/studios/DiffSynth-Studio/EliGen)
+* Training dataset: [ModelScope Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet)
+
+## Methodology
+
+![regional-attention](./samples/regional_attention.jpg)
+
+We introduce a regional attention mechanism within the DiT framework to effectively process the conditions of each entity. This mechanism enables the local prompt associated with each entity to semantically influence specific regions through regional attention. To further enhance the layout control capabilities of EliGen, we meticulously contribute an entity-annotated dataset and fine-tune the model using the LoRA framework. 
+
+1. **Regional Attention**: Regional attention is shown in above figure, which can be easily applied to other text-to-image models. Its core principle involves transforming the positional information of each entity into an attention mask, ensuring that the mechanism only affects the designated regions.
+   
+2. **Dataset with Entity Annotation**: To construct a dedicated entity control dataset, we start by randomly selecting captions from DiffusionDB and generating the corresponding source image using Flux. Next, we employ Qwen2-VL 72B, recognized for its advanced grounding capabilities among MLLMs, to randomly identify entities within the image. These entities are annotated with local prompts and bounding boxes for precise localization, forming the foundation of our dataset for further training.
+
+3. **Training**: We utilize LoRA (Low-Rank Adaptation) and DeepSpeed to fine-tune regional attention mechanisms using a curated dataset, enabling our EliGen model to achieve effective entity-level control.
+
+## Usage
+This model was trained using [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio). We recommend using DiffSynth-Studio for generation.
+```shell
+git clone https://github.com/modelscope/DiffSynth-Studio.git
+cd DiffSynth-Studio
+pip install -e .
+```
+1. **Entity-Level Controlled Image Generation**
+   EliGen achieves effective entity-level control results. See [entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control.py) for usage.
+2. **Image Inpainting**
+   To apply EliGen to image inpainting task, we propose a inpainting fusion pipeline to preserve the non-painting areas while enabling precise, entity-level modifications over inpaining regions.
+   See [entity_inpaint.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_inpaint.py) for usage.
+3. **Styled Entity Control**
+   EliGen can be seamlessly integrated with existing community models. We have provided an example of how to integrate it with the IP-Adpater. See [entity_control_ipadapter.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control_ipadapter.py) for usage.
+4. **Entity Transfer**
+   We have provided an example of how to integrate EliGen with In-Cotext LoRA, which achieves interesting entity transfer results. See [entity_transfer.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_transfer.py) for usage.
+5. **Play with EliGen using UI**
+   Download the checkpoint of EliGen from [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen) to `models/lora/entity_control` and run the following command to try interactive UI: 
+   ```bash
+   python apps/gradio/entity_level_control.py
+   ```
+## Examples
+### Entity-Level Controlled Image Generation
+
+1. The effect of generating images with continuously changing entity positions.
+
+<center>	
+    <video muted="true" autoplay="true" loop="true" height="512" width="512" src="./samples/video.mp4"></video>
+</center>
+
+1. The image generation effect of complex Entity combinations, demonstrating the strong generalization of EliGen. See [entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control.py) `example_1-6` for generation prompts.
+
+|Entity Conditions|Generated Image|
+|-|-|
+|![eligen_example_1_mask_0](./samples/e1_m.png)|![eligen_example_1_0](./samples/e1.png)|
+|![eligen_example_2_mask_0](./samples/e2_m.png)|![eligen_example_2_0](./samples/e2.png)|
+|![eligen_example_3_mask_27](./samples/e3_m.png)|![eligen_example_3_27](./samples/e3.png)|
+|![eligen_example_4_mask_21](./samples/e4_m.png)|![eligen_example_4_21](./samples/e4.png)|
+|![eligen_example_5_mask_0](./samples/e5_m.png)|![eligen_example_5_0](./samples/e5.png)|
+|![eligen_example_6_mask_8](./samples/e6_m.png)|![eligen_example_6_8](./samples/e6.png)|
+
+1. Demonstration of the robustness of EliGen. The following examples are generated using the same prompt but different seeds. Refer to [entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control.py) `example_7` for the prompts.
+
+|Entity Conditions|Generated Image|
+|-|-|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_5](./samples/e7_1.png)|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_6](./samples/e7_2.png)|
+![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_7](./samples/e7_3.png)|
+|![eligen_example_7_mask_5](./samples/e7_m.png)|![eligen_example_7_8](./samples/e7_4.png)|
+
+### Image Inpainting
+Demonstration of the inpainting mode of EliGen, see [entity_inpaint.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_inpaint.py) for generation prompts.
+|Inpainting Input|Inpainting Output|
+|-|-|
+|![inpaint_i1](./samples/inpaint_i1.jpg)|![inpaint_o1](./samples/inpaint_o1.png)|
+|![inpaint_i2](./samples/inpaint_i2.png)|![inpaint_o2](./samples/inpaint_o2.png)|
+### Styled Entity Control
+Demonstration of the styled entity control results with EliGen and IP-Adapter, see [entity_control_ipadapter.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_control_ipadapter.py) for generation prompts.
+|Style Reference|Entity Control Variance 1|Entity Control Variance 2|Entity Control Variance 3|
+|-|-|-|-|
+|![ip_ref](./samples/ip_ref.png)|![ip_1](./samples/ip_1.png)|![ip_2](./samples/ip_2.png)|![ip_3](./samples/ip_3.png)|
+
+We also provide a demo of the styled entity control results with EliGen and specific styled lora, see [./styled_entity_control.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/styled_entity_control.py) for details. Here is the visualization of EliGen with [Lego dreambooth lora](https://huggingface.co/merve/flux-lego-lora-dreambooth).
+|![image_1_base](./samples/styled_entity_control_example_1_mask_0.png)|![result1](./samples/styled_entity_control_example_2_mask_0.png)|![result2](./samples/styled_entity_control_example_3_mask_27.png)|![result3](./samples/styled_entity_control_example_4_mask_21.png)|
+|-|-|-|-|
+|![image_1_base](./samples/styled_entity_control_example_5_mask_0.png)|![result1](./samples/styled_entity_control_example_6_mask_8.png)|![result2](./samples/styled_entity_control_example_7_mask_5.png)|![result3](./samples/styled_entity_control_example_7_mask_6.png)|
+
+### Entity Transfer
+Demonstration of the entity transfer results with EliGen and In-Context LoRA, see [entity_transfer.py](https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/EntityControl/entity_transfer.py) for generation prompts.
+
+|Entity to Transfer|Transfer Target Image|Transfer Example 1|Transfer Example 2|
+|-|-|-|-|
+|![ic_logo](./samples/ic_logo.jpg)|![ic_target](./samples/ic_target.png)|![ic_1](./samples/ic_1.jpg)|![ic_2](./samples/ic_2.jpg)|
+
+## Citation
+If you find our work helpful, feel free to give us a cite.
+```
+@article{zhang2025eligen,
+  title={Eligen: Entity-level controlled image generation with regional attention},
+  author={Zhang, Hong and Duan, Zhongjie and Wang, Xingjun and Chen, Yingda and Zhang, Yu},
+  journal={arXiv preprint arXiv:2501.01097},
+  year={2025}
+}
+```

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"text-to-image-synthesis"}

model_bf16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2a253f6d7b40754140cf0ed48c99477057b15f44f4f3b40b4debd0e721363bf
+size 612742344

samples/video.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7906b41bcb415c11e6a65774e37332125f0b31fd8a0cb94ff0371e57f3323d64
+size 2952060