ControlNet - The Secret Sauce of AI Image Wizardry

Publish in  Generative Art, Stable Diffusion, Comfyui, Controlnet
 •   •  2 min read

Ok, so let's dive into some ComfyUI workflows, shall we? Time to peek behind the digital curtain! 🧙‍♂️

ControlNet - The Secret Sauce of AI Image Wizardry

During the weekends, I’ve been diving into ComfyUI workflows and hunting for some eye-popping imagery. I started yesterday with a non-ControlNet illustration (original). The original scene emerged from ArthemyComics SDXL model paired with a Moebius-style LoRA, giving it that distinctive cosmic-surrealist Moebius vibe. For the color scheme, I kept things neutral in the prompt with just that striking red as the contrast element (moving away from Moebius’s signature pastel palette) – just to create a visual punch.

Then today I decided to experiment with ControlNet. I wanted to recreate similar vibes using a different model without any LoRA support, just to see how much control I have over a generated image.

Initially, I deployed only the Depth preprocessor to maintain spatial awareness, but ran into a curious issue: those stubborn red grass patches kept appearing on the wrong side of the tunnel, not where the boy was standing, despite making this detail very clear in the prompt. The solution? Adding a Color preprocessor to the mix – this second layer of control tamed those rebellious red pixels after the second generation try.

Original (no ControlNet)
First attempt with Depth (ControlNet)
Depth refinement
Depth refinement - tweaking prompt
Color + Depth
Final: All three preprocessors (Depth+Color+Canny)

Then I duplicated the preprocessor nodes and deployed Canny as the final piece of the puzzle. This trio of preprocessors dramatically improved the AI’s understanding of the tunnel structure – those concentric rings sharpened up and the perspective went from “pretty cool” to “mind-bending portal to another dimension.”

ControlNet allows us to guide AI image generation with specific visual conditions like poses, depth maps, or line drawings, giving us precise control over the composition and structure of the final output while preserving artistic style. 💥 🥊

I’m making this workflow available here (just save it as a .json file and drag/drop to ComfyUI) for enthusiasts to experiment with.

What you’ll need to make it work:

That’s it. Dive in, tweak the knobs, and see where your own tunnel vision leads.