Following the emergence of revolutionary diffusion models such as DALLE-2, Imagen, and Stable Diffusion in the middle of 2022,  a significant breakthrough was introduced in February 2023 with the release of ControlNets. These innovative models bridged a crucial gap in the generative landscape: lack of a more explicit control over the generation results. ControlNets enhance pretrained large diffusion models by providing support for additional input conditions. With ControlNets, models like Stable Diffusion could be augmented, enabling the integration of conditional inputs such as edge maps, segmentation maps, depth maps, body pose maps, and more.

The inclusion of additional explicit controls proved to be a game-changer, significantly enhancing the editing, stylizing, and manipulation capabilities of user images while preserving their contents, scene layouts, and more. Empowered by this versatile tool, we have introduced a diverse range of innovative and highly popular AI-driven features into our apps, including AI-Selfies, AI-Scenes, AI-Rooms, AI-Anime, AI-Outfits, AI Cartoons, AI Comics and more. In addition to off-the-shelf ControlNet models, which prove to be a very important building block in our pipelines, we train our own ControlNet models for specific control modalities which are of special interest to Lightricks, for example for better retention of facial identity.

Our groundbreaking approach combines the generative inpainting capabilities of diffusion models, which allow for varying degrees of change in different areas of images, with the spatial controls offered by ControlNets. This powerful combination enables users to retain creative control while seamlessly adhering to the desired spatial conditions. Gone are the days of simple text-to-image conditioning; we harness the potential of diffusion models, inpainting techniques, spatial controls, and additional computer vision models such as semantic segmentation, saliency detection, and matting models. Together, these components form a very powerful stylization machine that achieves what was considered science fiction just a few months ago.