RFNet-Edit
LLM-assisted diffusion framework that turns editing instructions into executable multimodal plans for multi-object image edits.
Off-the-shelf diffusion inpainting struggles with multi-object edit instructions and tends to rewrite the background. The engineering bet was to stop treating the instruction as a single prompt and instead parse it into an executable, instance-level plan.
That means an LLM that emits structured edits, a detector-segmentor that finds each target, and a generator that inpainted only those regions — then a CLIP score that actually moves.
- Built the LLM-driven instruction parsing and segmentation stage: GPT-4o-mini emits a JSON edit plan; GroundingDINO localizes objects; SAM (ViT-H) produces masks.
- Evaluated semantic alignment on EditBench. RFNet-Edit reached a CLIP score of 27.41, a 6.67% improvement over the SDXL inpainting baseline (25.69).
- Generation and inpainting were implemented in the collaborator's Reality-and-Fantasy-Edit pipeline; this repository is the parse → detect → mask → evaluate half of the stack.
Perception
GroundingDINO for open-vocabulary localization, SAM for pixel-level masks. Mask quality was compared against EditBench ground truth — that comparison is the reason later inpainting can preserve background.
ML / Data
GPT-4o-mini decomposes a free-form instruction into a JSON schema of target objects, output objects, and actions. The generator consumes that plan rather than the raw sentence. CLIP score on EditBench is the reported metric.
Multi-object edits fail when the model has to jointly invent what to change and where. Splitting planning from execution is the whole method.
A CLIP gain is not a substitute for looking at masks. If SAM under-segments, SDXL will happily inpaint the sofa into the wall — the metric will not always catch it.