and accurately restore colors even when the line drawings undergo changes in pose and proportion. A similar project was introduced earlier:
Yihao Meng, Hao Ouyang, Hanlin Wang, etc., from HKUST, Ant Group, NJU, ZJU, HKU, and other universities and companies.
Abstract
Automatically coloring line drawings in videos plays a significant role in simplifying the animation production process and reducing labor costs. However, current automation attempts face the following challenges:
Character design images are not aligned with line-drawing sketches。 Need to maintain temporal consistency。
Traditional methods usually rely on manually colored keyframes and dense line-drawing guidance, which:
increase the workload of artists. due to non-binary line-drawing conditions causing color information leakage.
New method
that automatically completes coloring through the following innovative solutions:
to simplify the coloring process. and injection modules to accurately map the color information of reference images onto line drawings. to reduce the need for intermediate frame line drawings through keyframe interpolation.
Key improvements include:
as conditions to prevent color information leakage. Introducing data augmentation techniques to enhance training stability.
Functions and application scenarios
1. Flexible use of
: maintaining color consistency even in cases of large pose and proportion changes. 
2. Same line drawing, different reference images
adjust details (such as lighting and background) according to different reference images while maintaining character identity consistency. 
3. Sparse input line drawings
Through a two-stage training strategy, only the starting and ending frame line drawings are needed to generate smooth and coherent animations. 
4. Supports multiple characters
It can automatically distinguish between multiple characters in the reference image and correctly color each character, even if the poses, angles, or relative positions in the line drawings change significantly. 
5. Background style transfer
Transfer background styles from reference images to generate backgrounds in different styles. 
Limitations
Objects not present in the reference image
When line drawings contain objects not present in the reference image, the model struggles to determine appropriate colors and can only infer based on existing color information, leading to inaccurate coloring. 
Character clothing differences
When the character's clothing in the line drawings differs from that in the reference image (even though it is the same character), the model can only speculate based on the color scheme of the clothing in the reference image, potentially causing consistency issues. 
Trial
https://huggingface.co/spaces/fffiloni/AniDoc
Technology
AniDoc adopts a two-stage training strategy. In the dense line drawing training stage, matching keypoint pairs are explicitly extracted from the reference image and every frame of the training video, and point mapping is constructed to represent correspondence relationships. In the sparse line drawing training stage, intermediate frame line drawings are removed, and point trajectories are generated by interpolating matching points from the start and end frames, guiding the generation of intermediate frames.
Method comparison
AniDoc outperforms existing methods in the following aspects:
Character consistency. Background adaptation. Smoothness over time.