UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation: We propose UniGenBench++, a unified semantic benchmark for T2I generation. It supports both short and long prompts in Chinese and English, featuring a streamlined evaluation pipeline and a robust offline evaluation model.

Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning: We propose Pref-GRPO and UniGenbench, the first preference reward-based GRPO method for stable T2I reinforcement learning, and a unified T2I generation benchmark for fine-grained semantic consistency evaluation.

[NeurIPS 2025] Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning: We propose UnifiedReward-Think, the first unified multimodal CoT reward model.

Unified Reward Model for Multimodal Understanding and Generation: We release the UnifiedReward, the first unified reward model for multimodal understanding and generation assessment, enabling both pairwise ranking and pointwise scoring.

😊 Meta, Transition Matching: Scalable and Flexible Generative Modeling.

😊 NVIDIA, Stanford, Tsinghua, DiffusionNFT: Online Diffusion Reinforcement with Forward Process.

😊 University of California, USTC, PKU, BIGAI, MILR: Improving Multimodal Image Generation via Test-time Latent Reasoning.

😊 Kuaishou, Tsinghua, CUHK, Flow-GRPO: Training Flow Matching Models via Online RL.

😊 Tencent Hunyuan, MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE.

😊 Kling Team, CUHK MMLab, NJU, VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning.

😊 CUHK MMLab, Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO.

😊 Tencent Hunyuan X, X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again.

[2025/10/23] 🔥🔥🔥 We release UnifiedReward-Edit-[3b/7b], a unified reward model for both Text-to-Image and Image-to-Image generation trained on approximately 700K unified image generation and editing reward data!! For image editing reward task, our models support:

1. Pairwise Rank — directly judge which of two edited images is better.

2. Pairwise Score — assign a separate score to each image in a pair.

3. Pointwise Score — rate a single image on two axes: instruction-following and overall image quality.

🚀 The image editing reward inference code is available at UnifiedReward-Edit/ directory, while T2I inference code is unchanged from previous models. The editing training data is preprocessed from EditScore and EditReward and will be released soon. We sincerely appreciate all contributors!!

[2025/9/25] 🔥🔥🔥 We release UnifiedReward-2.0-qwen-[3b/7b/32b/72b]. This version introduces several new capabilities:

1. Pairwise scoring for image and video generation assessment on Alignment, Coherence, Style dimensions.

2. Pointwise scoring for image and video generation assessment on Alignment, Coherence/Physics, Style dimensions.

The added inference code is available at inference_qwen/UnifiedReward-2.0-inference directory. The newly added training data has been released here 😊.

😊 We are actively gathering feedback from the community to improve our models. We welcome your input and encourage you to stay updated through our repository!!

😊 We appreciate the mradermacher team for providing the GGUF version of our models, and the Tencent Hunyuan team for providing the evaluation results on several T2I models using UnifiedReward-qwen-7b!! The evaluation was conducted on 400 prompts sourced from here.

Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

We release UnifiedReward-Think -- the first unified multimodal CoT reward model, capable of multi-dimensional, step-by-step long-chain reasoning for both visual understanding and generation reward tasks.

Please refer to the README.md for training and inference details.

UnifiedReward-Think-qwen-2.0 [3b/7b/32b/72b] are coming soon!!

🔥🔥 We release UnifiedReward-Think-qwen-7b, a more powerful unified multimodal CoT reward model built upon UnifiedReward-qwen-7b!!!!

🔥🔥 We released Gradio for UnifiedReward-Think!

1. Clone this repository and navigate to the UnifiedReward folder:

git clone https://github.com/CodeGoat24/UnifiedReward.git
cd UnifiedReward

2. Install the inference package:

conda create -n unifiedreward python=3.10 -y
conda activate unifiedreward
pip install --upgrade pip  
pip install -e ".[train]"
pip install flash_attn==2.5.8 --no-build-isolation

For Qwen2.5-VL based UnifiedReward models, you should first install the inference packages as follows:

pip install git+https://github.com/huggingface/transformers accelerate qwen-vl-utils[decord]==0.0.8

We provide reference pair ranking and point score inference code for each task in the ./inference and ./inference_qwen directories.

inference
├── image_generation                  
    ├── pair_rank_image_generation.py            
    └── point_score_image_generation.py         
├── video_understanding                 
    ├── pair_rank_video_understanding.py            
    └── point_score_video_understanding.py
... 

Note that our model is not constrained to a fixed input prompt style. You can flexibly adjust inputs based on your requirements.

We provide vLLM inference code for UnifiedReward-qwen in vllm_qwen directory.

1. Install vLLM

pip install vllm==0.9.0.1 transformers==4.52.4

2. Deploy vLLM Server

bash vllm_qwen/vllm_server.sh

3. Inference Request to vLLM Server

python vllm_qwen/vllm_inference.py

We provide SGLang inference code for UnifiedReward-llava in sglang_llava directory.

1. Install SGLang

pip install "sglang[all]"

2. Deploy SGLang Server

bash sglang_llava/sglang_server.sh

3. Inference Request to SGLang Server

python sglang_llava/sglang_inference.py

We use LLaMA-Factory to train the SFT model.

1. Clone the LLaMA-Factory repository and install the dependencies.

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"

Follow this README (Multimodal Image Dataset) to prepare our released datasets.

2. Run the following command to train the SFT model.

llamafactory-cli train examples/train_full/qwen2_5vl_full_sft.yaml

Please download our constructed unified preference dataset from Huggingface and put it in ./dataset/.

dataset
├── EvalMuse                  
    ├── pairwise            
    └── pointwise
    └── ...            
└── HPD                   
└── LiFT-HRA
└── LLaVA-Critic 
    ├── pairwise            
    └── pointwise
    └── ...
└── OIP
└── ShareGPTVideo
    ├── pairwise            
    └── pointwise
    └── ...      
└── VideoDPO 
└── VideoFeedback
└── train_data.yaml

bash train.sh

[
    {
    "prompt": "",
    "image": "",
    },
    ...
]

# image understanding 
cd preference_data_construction/image_understanding
python infer+sift.py # you need to fill the 'image_folder' and 'data_path' in this file

# video understanding 
cd preference_data_construction/video_understanding
python infer+sift.py # you need to fill the 'image_folder' and 'data_path' in this file

[
    {
    "id": "",
    "image": "",
    "prompt": "",
    "chosen": "",
    "rejected": ""
    },
    ...
]

# image understanding 
bash dpo_image_understand_ov7b.sh 

# video understanding 
bash dpo_video_understand_llava_video_7b.sh

cd DiffusionDPO
conda create -n diffdpo python=3.10 -y
conda activate diffdpo
pip install -r requirements.txt

[
    {
    "prompt": "",
    },
    ...
]

python data_generation.py # you need to fill the 'data_path' in this file

python sift_dpo_data.py

[
    {
        "id": "",
        "caption": "",
        "jpg_0": "", #chosen image path
        "jpg_1": "", #rejected image path
        "label_0": 1,
    },
    ...
]

bash launchers/turbo_dpo.sh

cd VideoDPO
conda create -n videodpo python=3.10 -y
conda activate videodpo
pip install -r requirements.txt

mkdir -p checkpoints/vc2
wget -P checkpoints/vc2 https://huggingface.co/VideoCrafter/VideoCrafter2/resolve/main/model.ckpt

[
    {
    "prompt": "",
    },
    ...
]

bash data_generation.sh # you need to fill '--prompts_file' in this file

python sift_dpo_data.py

[
    {
        "id": "",
        "caption": "",
        "chosen": "", # chosen video path
        "rejected": "", # rejected video path
    },
    ...
]

bash run.sh

We provide several evaluation code in ./benchmark_evaluation directory.

We provide evaluation code for GenAI-Bench-Video, GenAI-Bench-Image, VideoGen-RewardBench and VL-RewardBench benchmarks.

We provide evaluation code for MSRVTT, MSVD, and TGIF benchmarks while using the VLMEvalKit toolkit for evaluating LongVideoBench, MLVU, and Video-MME benchmarks with 64 input frames.

We use LMMs-Eval toolkit to evaluate LLaVABench, WildVision, LLaVABench-Wilder, LiveBench, and MMHal benchmarks.

We utilize the image reward model, i.e., PickScore, HPS and ImageReward for quality assessment.

VBench is used for video generation assessment.

If you have any comments or questions, please open a new issue or feel free to contact Yibin Wang.

In this work, reward model and image/video understanding DPO code is based on LLaVA-Next, while image and video generation DPO is based on DiffusionDPO and VideoDPO.

We also utilize LMMs-Eval and VLMEvalKit toolkits for evaluation.

Thanks to all the contributors!

@article{unifiedreward-think,
  title={Unified multimodal chain-of-thought reward model through reinforcement fine-tuning},
  author={Wang, Yibin and Li, Zhimin and Zang, Yuhang and Wang, Chunyu and Lu, Qinglin and Jin, Cheng and Wang, Jiaqi},
  journal={arXiv preprint arXiv:2505.03318},
  year={2025}
}

@article{unifiedreward,
  title={Unified reward model for multimodal understanding and generation},
  author={Wang, Yibin and Zang, Yuhang and Li, Hao and Jin, Cheng and Wang, Jiaqi},
  journal={arXiv preprint arXiv:2503.05236},
  year={2025}
}