Multimodal generative AI for conceptual design: enabling text-based and sketch-based human-AI conversations

Abstract

Recent advances in AI offer promising opportunities for creative design, particularly through the generation of inspirational images. While prior research has explored the general benefits and limitations of text-to-image tools, there is significant potential in overcoming these constraints by investigating agile, multimodal prompting to facilitate more project-appropriate human-AI interaction. We present the development of a system designed to support both text-based and sketch-based image generation, serving as a research artefact for studying creativity support through multimodal Generative AI. The system enables dynamic dialogue interaction and visualization of the respective contributions. This paper focuses on the development of this AI system as a research artefact to enable future research through design, exploring how multimodal prompting can influence the design process.