GIE-Bench: Towards Grounded Evaluation for Text-Guided Image Editing

Summary

The article introduces GIE-Bench, a new benchmark for evaluating text-guided image editing models, focusing on functional correctness and image content preservation. It highlights the limitations of existing evaluation methods that rely on image-text similarity metrics, proposing a more grounded approach through automatically generated questions and object-aware masking techniques. The benchmark comprises over 1000 editing examples across various content categories, providing a comprehensive framework for assessing model performance. The study compares the latest model, GPT-Image-1, against other state-of-the-art models, revealing trade-offs in instruction-following accuracy and the tendency to over-modify irrelevant image regions.

Key Learnings

• 1GIE-Bench offers a structured evaluation framework that enhances the accuracy of assessing text-guided image editing models.
• 2The benchmark's dual focus on functional correctness and content preservation addresses critical gaps in current evaluation methodologies.
• 3GPT-Image-1 demonstrates superior instruction-following capabilities but struggles with maintaining visual consistency in non-targeted regions, indicating a trade-off in model performance.
• 4The inclusion of detailed editing instructions and spatial object masks in the benchmark allows for a more nuanced evaluation of model outputs.

Who Should Read This

Senior AI Researchers specializing in Computer Vision and Generative AI looking to enhance evaluation methodologies for image editing models.

Test Your Knowledge

Topics