Effectiveness of Model Tailor patches in complex and open-domain settings

Establish the effectiveness of Model Tailor’s extracted and decorated sparse parameter patches for mitigating catastrophic forgetting in more complex and open-domain multimodal tasks beyond the evaluated benchmarks.

Background

Model Tailor identifies a minimal set of finetuned parameters (a "patch") and decorates them to balance performance on current and prior tasks, showing improvements on VQA and image captioning with InstructBLIP and LLaVA-1.5.

The authors note that validation to date is limited and the method’s effectiveness remains to be demonstrated in broader, more complex domains.