Scalable Structured Variational Sparse Tensor Representation Learning for Distributed Multimodal Data Streams
- College of Mathematics and Computer, Jilin Normal University, Siping 136000, China
Abstract
Representation learning for distributed multimodal data streams is an important task in intelligent healthcare, mobile sensing, human activity recognition, and edge-assisted monitoring. Its goal is to learn stable, scalable, and uncertainty aware representations from continuously arriving multimodal data distributed across multiple clients. However, existing federated learning and multimodal fusion meth ods often rely on complete modality availability and a shared input structure across clients, making them less suitable for scenarios with client-level modality incon sistency, asynchronous modality arrival, and highly uncertain observations. In ad dition, many existing approaches mainly depend on parameter averaging, feature reconstruction, or missing-modality generation, which makes it difficult to capture high-order cross-modal dependencies and posterior uncertainty in distributed envi ronments. To address these challenges, this paper proposes a Scalable Structured Variational Sparse Tensor framework, abbreviated as SVST. Specifically, SVST uses sparse low-rank tensor factors to directly generate the natural parameters of a structured variational posterior, enabling high-order multimodal interactions and uncertainty correlations to be modeled during posterior construction. Meanwhile, each client uploads only compact natural-parameter increments and sparse factor updates, and the server performs aggregation in the natural-parameter space. This design avoids raw-data transmission, full neural-parameter averaging, and dense global tensor construction. Furthermore, SVST introduces a missing-modality-aware online ELBO update, delay-aware weighting, and uncertainty-confidence regula tion to adapt to incomplete and asynchronous multimodal streams. Experiments on distributed multimodal datasets, including MIMIC, CMU-MOSEI, and OPPORTU NITY, show that SVST achieves overall superior or comparable performance com pared with existing federated learning, multimodal fusion, and multimodal vari ational inference baselines in terms of predictive performance, calibration error, communication cost, and client-level stability. Further robustness analysis indicates that, under increasing modality missingness, stronger asynchronous delay, and non IID client partitions, the proposed SVST effectively reduces performance degrada tion and improves prediction reliability and model robustness. These results demon strate that SVST provides a practical and robust solution for representation learning in distributed multimodal data stream tasks.
Key words
Distributed Multimodal Data Streams, Structured Variational Inference, Sparse Tensor Factorization, Natural-parameter Aggregation, Missing-modality Learn ing
Journal's Facebook page