I watched the commentary version of "The Crime Sketch" during the day, and in one episode, the protagonist's master was trapped in a difficult situation due to AI video fraud. The key to solving it lay in the light issue - the unnatural light and shadow in the video became the main basis for identifying AI synthesis.
. It is an interactive light source drawing tool under development, the full name is
Demo
Currently, although this tool is still under development, its Demo version has been launched - https://huggingface.co/spaces/lllyasviel/LuminaBrush.
Try it out
Technical framework
This project uses a two-stage method to allow users to directly "draw" lighting effects on images:
The first step in the framework is to convert the input image into a "uniform lighting" effect - similar to a scene illuminated by a white, evenly distributed environmental light source. The core of this step lies in stripping away the complex lighting information in the image, providing a clean base for subsequent reshaping of the lighting.
After completing the uniform lighting processing, the second step is to generate realistic lighting effects based on user-drawn "light source lines." This step-by-step approach avoids potential complexity constraint issues that may arise from trying to achieve everything in one step (such as consistent light transmission), making the model learning more direct and effective.
The LuminaBrush team found during design that using "uniform lighting" images as an intermediate representation can avoid problems brought by traditional 3D reflectance maps, such as overly sharp grid boundaries or insufficient surface smoothness. Moreover, it retains details while being suitable for handling complex textures, such as skin texture, hair, and fur.
The team first collected a small set of "uniform lighting" images as the initial training set and used Flux LoRA (a lightweight fine-tuning method) to generate more similar images. Then, random normal vectors were generated to re-light these "uniform lighting" images for training, enabling the model to learn how to extract "uniform lighting" effects from any input image. Subsequently, they extracted "uniform lighting" features from a large number of high-quality real-world scene images to build paired data, further training the final interactive lighting generation model.
It's worth noting that this "uniform lighting" processing is not only an intermediate step in generating lighting but can also be independently applied to "de-lighting" tasks, helping users optimize the light distribution in images.
THE END
Finally, here are a few other studies I shared earlier by the author of ControlNet: