Order Matters: Permutation-based Prompt Optimization for Personalized Image Generation

Abstract

Since AI-assisted content generation has received due attention of late, prompt reordering has emerged as a promising method for prompt tuning. This paper presents a framework named Interactive Prompt Permutation Optimization (IPPO) for text-to-image generation, which optimizes the order of prompt words according to user preferences. IPPO integrates user feedback into the optimization process in order to enhance personalization as well as reduce costs. We design a surrogate model in concert with an ordering matrix, which efficiently estimates user preferences from sparse feedback from the user. Furthermore, we develop a Permutation Genetic Algorithm (PermGA) equipped with a novel repairing operator derived from the surrogate model's dominance information in order to further enhance the efficiency of optimization. A set of experiments on the Linear Ordering Problem benchmark demonstrated the fundamental effectiveness of our approach. We also conducted experiments on image generation in order to verify the framework's validity in reflecting user preferences, achieving competitive results in terms of image quality and relevance compared to SOTA methods. © 2013 IEEE.