LoRA Captioner

A command-line tool to automatically caption images for LoRA training datasets using Vision Language Models (VLLMs).

Features

• Ultra-Detailed Captions: Uses Qwen3-VL-2B-Instruct for rich, comprehensive image descriptions optimized for LoRA training
• Multiple LoRA Types: Optimized captioning strategies for Character, Style, Concept, Outfit, Scene, Generic, Background, and Trigger-only LoRAs
• Generic Mode: <shot type> of <broad subject class> in <detailed background> — framing variety without leaking subject detail
• Background Mode: Detailed environment-only captions — no subject, no shot type, no camera framing
• Trigger-Only Mode: Skip the LLM entirely — caption files contain only the trigger word, ideal for simple activation LoRAs
• Multiple Model Options: Choose between Qwen3-VL (default), Florence-2, or BLIP models
• Automatic Model Download: Downloads the VLLM model on first run
• Smart Device Detection: Automatically selects GPU or CPU based on available hardware
• Batch Renaming: Renames images to consistent dataset format

Installation

Prerequisites

1. Python 3.10 or higher - Download from python.org

   • During installation, check "Add Python to PATH"

2. PyTorch (recommended for GPU acceleration):

   # For CUDA 11.8
   pip install torch --index-url https://download.pytorch.org/whl/cu118
   
   # For CUDA 12.1
   pip install torch --index-url https://download.pytorch.org/whl/cu121
   
   # For CPU only
   pip install torch
   

Install from PyPI (coming soon)

pip install lora-captioner

Install from Source

git clone https://github.com/jmpijll/lora-captioner.git
cd lora-captioner
pip install -e .

Verify Installation

lora-captioner --help

Quick Start

# Caption images for a character LoRA
lora-captioner -i ./my_images -n "my_character" -t character -w "sks_person"

# Caption images for a style LoRA
lora-captioner -i ./style_images -n "moebius_style" -t style

# Caption images for a concept LoRA
lora-captioner -i ./concept_images -n "special_hat" -t concept

# Trigger-only mode: just the trigger word, no LLM needed
lora-captioner -i ./trigger_images -n "activation" -t trigger -w "ohwx_person"

# Generic mode: short, non-specific captions (no colors, names, or fine details)
lora-captioner -i ./mixed_images -n "general_dataset" -t generic

# Background mode: detailed environment description only (no subject, no shot type)
lora-captioner -i ./scene_images -n "scene_dataset" -t background

Usage

lora-captioner [OPTIONS]

Options:
  -i, --input PATH          Input folder containing images [required]
  -n, --dataset-name TEXT   Name for the dataset (used in file renaming) [required]
  -t, --lora-type TYPE      Type of LoRA: character, style, concept, outfit, scene, generic, background, or trigger [required]  
  -w, --trigger-word TEXT   Trigger word to prepend to captions
  -o, --output PATH         Output folder (default: same as input)
  --device TEXT             Device to use: auto, cuda, or cpu (default: auto)
  --model TEXT              Model to use: qwen (default), florence, or blip
  --no-rename               Skip renaming images
  --recursive               Search for images in subdirectories
  --dry-run                 Preview actions without making changes
  --version                 Show version and exit
  --help                    Show this message and exit

LoRA Type Strategies

Character LoRA

• Captions describe pose, expression, background, lighting
• Does NOT describe permanent character features (hair color, signature outfit)
• Use trigger word to identify the character

Style LoRA

• Can use tag-based or natural language captions
• Best with diverse training images
• Trigger word identifies the style

Concept LoRA

• Captions describe context and environment
• Does NOT describe the core concept being trained
• Trigger word identifies the concept

Trigger-Only LoRA

• Caption files contain only the trigger word — no LLM inference at all
• Perfect for simple activation LoRAs where you just want to associate a token with a set of images
• No model download needed, runs instantly
• Every image gets the same caption: ohwx_person (or whatever trigger word you set)
• If no trigger word is provided, caption files will be empty

Generic LoRA

• Output: <shot type> of <subject class> in <detailed background>
• Shot type is 1–3 words of cinematography terminology (wide shot, close-up, overhead shot, Dutch angle, etc.)
• Subject class is broad only (a woman, a man, a person, a group, a landscape, etc.) — no hair, no face, no clothing, no age, no expression, no pose
• Background is described in high detail — materials, textures, lighting, depth of field, atmosphere, props, architecture, weather, signage
• Useful for general-purpose datasets where you want framing variety and rich environment detail without leaking any subject-specific detail
• Uses <CAPTION> on Florence-2, the standard a photo of prompt on BLIP, and a dedicated system prompt on Qwen3-VL (Qwen3-VL is the only model that reliably follows the format)

Background LoRA

• One flowing paragraph (3–6 sentences) describing the environment only — high detail on materials, textures, lighting, depth, atmosphere, props, architecture, weather, signage
• No subject description (no people, no animals, no main foreground object)
• No shot type (no "wide shot", "close-up", etc.)
• No camera terms (no aperture, focal length, lens, angle)
• Useful when you want rich environment captions without leaking any subject detail — pairs naturally with generic mode if you want both separately
• Uses <CAPTION> on Florence-2, the standard a photo of prompt on BLIP, and a dedicated system prompt on Qwen3-VL (Qwen3-VL is the only model that reliably follows the format)

Requirements

• Python 3.10+
• 4GB RAM minimum (CPU mode)
• 2GB VRAM minimum (GPU mode)

Model Information

This tool supports three captioning models:

Qwen3-VL (Default)

• Model: Qwen/Qwen3-VL-2B-Instruct
• Pros: Ultra-detailed captions with comprehensive descriptions including textures, lighting, composition, and camera perspective
• Best for: High-quality LoRA training datasets requiring rich detail
• Requirements: ~4GB VRAM (GPU) or 8GB RAM (CPU)

lora-captioner -i ./images -n "dataset" -t character

Florence-2

• Model: microsoft/Florence-2-large
• Pros: Detailed captions, good balance of speed and quality
• Best for: General-purpose captioning

lora-captioner -i ./images -n "dataset" -t character --model florence

BLIP (Fallback)

• Model: Salesforce/blip-image-captioning-large
• Pros: Stable, lightweight, simpler captions
• Use when: You need shorter captions or have limited resources

lora-captioner -i ./images -n "dataset" -t character --model blip

Documentation

• VLLM Model Comparison
• Captioning Strategies
• System Prompts Reference

License

MIT License - see LICENSE for details.

Contributing

Contributions are welcome! Please read the AGENTS.md file for development guidelines.