Abstract: We considered six Lagrangian based optimization algorithms that can be implemented in a distributed way and used for solving separable problems. These algorithms are; the Lagrangian [Arrow K.J. et al (1958)], the Multiplier Methods [Hestenes M. A. (1969), Powell, M. J. D. (1969)], Alternating Multiplier Method (ADMM) [Daniel G A et al (1976)], and the algorithms proposed by Bertsekas [Bertsekas D. P.l (1979)], Tanikawa-Mukai [Tanikawa A et al (1985)], Tatjewski [Tatjewski P (1989)], Arnold et al [Arnold E et al (1994)], and they were implemented in MATLAB. We proposed a decomposition strategy for model fitting problems and performed experiments on four convex (Lasso and Ridge Regressions, Support Vector Machine Classifier, and Logistic Regression Classifier) and one non-convex (Subset Selection) with publicly available datasets.

Keywords: Lagrangian; Optimization; Machine Learning; Alternating Direction Method of Multiplier; Separable Problems

Arnold E. and Tatjewski P. and Wo lochowicz P. (1994). Two methods for large-scale nonlinear optimization and their comparison on a case study of hydropower optimization. Journal of Optimization Theory and Applications, 81(2), 221-248, https://doi.org/10.1007/bf02191662.
Arrow K.J., Azawa H., Karreman M. R. C., Hurwicz L., Uzawa H., Chenery H.B. (1958). Studies in Linear and Non-linear Programming. Stanford University Press, 166-176.
Bertsekas D. P. (1979). Convexification procedures and decomposition methods for nonconvex optimization problems. Journal of Optimization Theory and Applications, 29(2), 169-197.
Daniel G. and Bertrand M. (1976). A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications, 2(1), 17-40, https://doi.org/10.1016/0898-1221(76)90003-1.
Hestenes M. A. (1969). Multiplier and gradient methods. Journal of Optimization Theory and Applications, 4(5), 303-320, https://doi.org/10.1007/bf00927673.
POWELL, M. J. D. (1969). A method for nonlinear constraints in minimization problems. Optimization, 83-298, https://ci.nii.ac.jp/naid/20000922074/en/.
Tanikawa A. and Mukai, H. (1985). A new technique for nonconvex primal-dual decomposition of a large-scale separable optimization problem. IEEE Transactions on Automatic Control, 30(2), 133-143, https://doi.org/10.1109/TAC.1985.1103899.
Tatjewski P. (1989). New dual-type decomposition algorithm for nonconvex separable optimization problems. Automatica, 25(2), 233-242, https://doi.org/10.1016/0005-1098(89)90076-9.