Multi-task ML

July 14, 2021 — July 14, 2021

edge computing
machine learning
neural nets
regression
sparser than thou
spatial
stochastic processes
time series
Figure 1

On training a model which predicts several things at once.

This is a very ML way of phrasing things. In classical statistics, if we have fit some multivariate regressions by a likelihood-based procedure, then it produces multivariate output. No problem. However, in machine learning, we frequently fit based on univariate predictive loss, and it is not clear, or at least affordable, to translate these univariate predictions to multivariate ones without starting over and simply training lots of univariate prediction models. In that context, it is not foolish to ask about multivariate predictions and think of them as the task of developing a “multi-task model” as some kind of new thing.

1 Weighted sum of multiple objectives

See multi-objective optimization.

2 Multi-task GPs

It is fairly natural to make a Gaussian process into a multivariate method; See Vector GP regression.

3 Incoming

4 References

Akkus, Chu, Djakovic, et al. 2023. “Multimodal Deep Learning.”
Bonilla, Chai, and Williams. 2007. “Multi-Task Gaussian Process Prediction.” In Proceedings of the 20th International Conference on Neural Information Processing Systems. NIPS’07.
Caruana. 1998. “Multitask Learning.” In Learning to Learn.
Dai, and Barber. 2016. “The Knockoff Filter for FDR Control in Group-Sparse and Multitask Regression.” arXiv Preprint arXiv:1602.03589.
Evgeniou, Micchelli, and Pontil. 2005. “Learning Multiple Tasks with Kernel Methods.” Journal of Machine Learning Research.
Evgeniou, and Pontil. 2004. “Regularized Multi-Task Learning.” In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’04.
Kong. 2019. “Dominantly Truthful Multi-Task Peer Prediction with a Constant Number of Tasks.” arXiv:1911.00272 [Cs, Econ].
Moreno-Muñoz, Artés-Rodríguez, and Álvarez. 2019. “Continual Multi-Task Gaussian Processes.” arXiv:1911.00002 [Cs, Stat].
Osborne, Roberts, Rogers, et al. 2008. “Towards Real-Time Information Processing of Sensor Network Data Using Computationally Efficient Multi-Output Gaussian Processes.” In 2008 International Conference on Information Processing in Sensor Networks (Ipsn 2008).
Platt, and Barr. 1987. “Constrained Differential Optimization.” In Proceedings of the 1987 International Conference on Neural Information Processing Systems. NIPS’87.
Radford, Wu, Child, et al. 2019. “Language Models Are Unsupervised Multitask Learners.”
Titsias, and Lázaro-Gredilla. 2011. “Spike and Slab Variational Inference for Multi-Task and Multiple Kernel Learning.” In Advances in Neural Information Processing Systems 24.
Williams, Klanke, Vijayakumar, et al. 2009. “Multi-Task Gaussian Process Learning of Robot Inverse Dynamics.” In Advances in Neural Information Processing Systems 21.