Determine the extent of LLM assistance for cross-modal alignment

Determine to what extent large language models assist cross-modal alignment in federated biomedical image classification, particularly for datasets requiring localized detection.

Background

FedSSMCoOp is designed to eliminate the need for an external LLM by using a Mamba Interaction Block to capture and align visual and textual features. The ablation experiments nevertheless indicate that incorporating an LLM at the server can improve performance on the Kvasir and RetinaMNIST datasets, which contain localized detection tasks. The paper therefore leaves unresolved how much an LLM contributes to cross-modal alignment and under what circumstances its use is beneficial in federated biomedical image classification.

References

Secondly, there were instances where the LLM presence in the server side of the federated learning aided in the performance especially in Kvasir and RetinaMNIST dataset due to their presence of localized detections. The forthcoming works can also focus more on whether to what extent the LLM help to aid these alignments.

— FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification  (2610.09907 - Das et al., 7 Oct 2026) in Section 6, Conclusion and Future Directions