Raghavan Muthuregunathan

This talk addresses the challenges faced by non-English users when interacting with text-to-image diffusion models and introduces a novel approach called Translation Augmented Generation. The predominance of English in training data has resulted in suboptimal performance for non-English prompts in image generation tasks, often producing culturally incongruent or irrelevant images. The authors present a prompt engineering technique that utilizes existing Large Language Models (LLMs) to detect language, translate, and augment non-English prompts with additional metadata. This method aims to enhance the quality and cultural relevance of generated images without requiring expensive model fine-tuning or building language-specific diffusion models from scratch. The article demonstrates the effectiveness of Translation Augmented Generation in improving image outputs for prompts in low-resource languages, showing how it captures both semantic meaning and cultural nuances. While acknowledging some limitations, such as increased token processing, the authors propose this technique as a promising solution to bridge the language gap in text-to-image generation, potentially democratizing access to AI-generated imagery across diverse linguistic communities.