This summer by the sea, I created a series of photographs. For this project, I gathered stones, seashells, sticks, and bits of glass on the beach, arranging them into compositions that resembled real food. Project idea: to train a neural network on these photographs and see what «menu» it comes up with on its own.
Source photographs for instruction
prompt:
a fruit plate with berries and sea glass leaves, everything made of stones with rough natural stone texture, small round red, orange, and deep purple pebbles as berries, green sea glass leaves, a large flat grey stone plate
Fruit Salad
prompt:
a breakfast with fried eggs, toast, and tomatoes, everything crafted from stones with a rough, natural stone texture, white pebbles serving as egg whites, a golden ochre round stone as the yolk, flat sandy beige stones as toast, and terracotta red stones as tomatoes with green sea glass leaves
Scrambled Eggs with Tomatoes and Toast
prompt:
a cup of tea with cookies, everything made of stones with a rough, natural stone texture, a small hollow grey stone cup, a flat round brown stone saucer, three round ochre and beige flat stones as cookies, a driftwood spoon
Cookie with Tea
prompt:
a birthday cake with candles, everything made of stones with a rough, natural stone texture, stacked round flat stones in cream white and pale pink, small red pebbles on top, thin white driftwood sticks as candles, grey stone plate
Strawberry Cream Cake
prompt:
a burger with fries, everything made of stones with a rough, natural stone texture, two flat, golden-brown stones as the bun, a dark brown stone patty, a green sea glass lettuce leaf, thin pale driftwood sticks as fries, grey stone plate
Burger with french fries
prompt:
a pizza on a round stone board, everything made of stones with a rough natural stone texture, a flat round ochre stone as the crust, small red terracotta pebbles as pepperoni, cream white pebbles as cheese, green sea glass as basil
Basil Pizza
prompt:
a bowl of soup with a spoon, everything made of stones with a rough, natural stone texture, a round hollow gray stone bowl filled with small orange and cream pebbles, a flat stone spoon, a green sea glass herb leaf
Vegetable Soup
prompt:
a plate of sushi rolls with driftwood chopsticks, everything made of stones with rough natural stone texture, round reddish-brown and cream white pebbles, grey flat stone plate.
Sushi set
Evaluation of Results
— The texture of the stones, sticks, and glass is excellently conveyed. There are no strange objects of unnatural origin.
— The stone compositions truly resemble their real prototypes.
— Occasionally, the neural network overdid the quantity of sticks (cutlery) around the plates.
— The natural, existing colors of the stones are well rendered.
— The recognizability of the dish depends on its shape: solid dishes (omelet, pizza, burger) read well. Liquid and amorphous dishes (soup, salad) simply turn into a pile of stones.
— In my source material, the «dishes» are minimalist, consisting of 5–8 stones. However, the model created 10–20 stones, resulting in more cluttered images.
The Learning Process
I fine-tuned the Stable Diffusion XL model using the DreamBooth LoRA method. This method does not retrain the entire neural network; it merely adds a small layer incorporating my style. The dataset consisted of 42 photographs of my stone compositions, cropped to a 1:1 square centered on the image.
For some of the photos, I provided my own descriptions. Initially, the BLIP model generated them automatically, but it failed to recognize that the stones depicted food (e.g., «a rock with a small piece of wood and a small piece of glass»). Therefore, I manually described the «dishes» for 15 photos and added the trigger phrase «top-down photo of a stone food composition in KAMNI style» to all captions.
Training was conducted on a free T4 GPU in Google Colab: 500 steps, taking about 50 minutes. The model viewed each photo approximately 72 times. I saved checkpoints at 250 and 500 steps to compare the intermediate and final results.
Parameters
— Base Model: Stable Diffusion XL base 1.0; VAE: sdxl-vae-fp16-fix; Captions: Salesforce/blip-image-captioning-base.
— Data: 42 photos, 1024×1024 during preparation, training at 512 resolution.
— Captions: BLIP for all, manual for 15.
— Steps: 500; batch size 2 with gradient accumulation 3 (total of 6 photos per step); learning rate 1e-4.
— Memory Saving: 8-bit Adam, gradient checkpointing, mixed precision fp16.
— Time: 50 minutes 47 seconds, approximately 6 seconds per step.
— Checkpoints: checkpoint-250 and checkpoint-500.
Notebook
