[1] Objective
Explore the capabilities of generative neural networks and create a series of images inspired by the illustrations of American children’s author and animator Dr. Seuss.
Currently, Dr. Seuss is the best-selling children’s author in English; many students learn to read through his books. When I was seven, I learned English using his tongue twisters and redrew the illustrations. While a few of his books have been translated and published in Russian, most are written in verse, so Russian-speaking audiences are primarily familiar with Dr. Seuss’s work through modern animated adaptations of his books.
[2] Stages of Work
2.1
— Checking GPU availability via nvidia-smi — Installing necessary libraries: bitsandbytes, transformers, diffusers, peft (for parameter tuning, including LoRA), and the Dreambooth-LoRA training script.
2.2
— Data set compilation from Dr. Seuss illustration images. I compiled a set of 54 square-format images and uploaded them to Google Colab in a separate directory.
2.3
— A prefix, «painting in the style of WYATT, ” is added to every image caption using the BLIP model. File paths and captions are saved in metadata.jsonl.
2.4
— Using the Dreambooth-LoRA technique for fine-tuning Stable Diffusion.
BASE MODELS:
stabilityai/stable-diffusion-xl-base-1.0
VAE madebyollin/sdxl-vae-fp16-fix
CORE TRAINING PARAMETERS:
Resolution: 512×512
Batch size: 2
Learning rate: 1e-4
Number of steps: 500
8-bit Adam (memory optimization)
FP16
— Running the training via accelerate
[3] Image Generation
After completing the course, I use the DiffusionPipeline to generate images based on various prompts. Generation settings: 25 steps, FP16.
[4] Project Outcomes
The generative model performs well when trained on a curated set of images and reproduces the illustrative style with high quality.