Abstract: Abstract We consider the stochastic variational inequality problem in which the map is expectation-valued in a component-wise sense. Much of the available convergence theory and rate statements for stochastic approximation schemes are limited to monotone maps. However, non-monotone stochastic variational inequality problems are not uncommon and are seen to arise from product pricing, fractional optimization problems, and subclasses of economic equilibrium problems. Motivated by the need to address a broader class of maps, we make the following contributions: (1) we present an extragradient-based stochastic approximation scheme and prove that the iterates converge to a solution of the original problem under either pseudomonotonicity requirements or a suitably defined acute angle condition. Such statements are shown to be generalizable to the stochastic mirror-prox framework; (2) under strong pseudomonotonicity, we show that the mean-squared error in the solution iterates produced by the extragradient SA scheme converges at the optimal rate of \({{\mathcal {O}}}\left( \frac{1}{{K}}\right) \) , statements that were hitherto unavailable in this regime. Notably, we optimize the initial steplength by obtaining an \(\epsilon \) -infimum of a discontinuous nonconvex function. Similar statements are derived for mirror-prox generalizations and can accommodate monotone SVIs under a weak-sharpness requirement. Finally, both the asymptotics and the empirical rates of the schemes are studied on a set of variational problems where it is seen that the theoretically specified initial steplength leads to significant performance benefits. PubDate: 2019-09-10

Abstract: Abstract This paper deals with a new variant of the inexact restoration method of Fischer and Friedlander (Comput Optim Appl 46:333–346, 2010) for nonlinear programming. We propose an algorithm that replaces the monotone line search performed in the tangent phase by a non-monotone one, using the sharp Lagrangian as merit function. Convergence to feasible points satisfying the convex approximate gradient projection condition is proved under mild assumptions. Numerical results on representative test problems show that the proposed approach outperforms the monotone version when a suitable non-monotone parameter is chosen and is also competitive against other globalization strategies for inexact restoration. PubDate: 2019-09-09

Abstract: Abstract Despite the vast literature on DRS and ADMM, there has been very little work analyzing their behavior under pathologies. Most analyses assume a primal solution exists, a dual solution exists, and strong duality holds. When these assumptions are not met, i.e., under pathologies, the theory often breaks down and the empirical performance may degrade significantly. In this paper, we establish that DRS only requires strong duality to work, in the sense that asymptotically iterates are approximately feasible and approximately optimal. PubDate: 2019-09-07

Abstract: Abstract We propose two limited-memory BFGS (L-BFGS) trust-region methods for large-scale optimization with linear equality constraints. The methods are intended for problems where the number of equality constraints is small. By exploiting the structure of the quasi-Newton compact representation, both proposed methods solve the trust-region subproblems nearly exactly, even for large problems. We derive theoretical global convergence results of the proposed algorithms, and compare their numerical effectiveness and performance on a variety of large-scale problems. PubDate: 2019-09-05

Abstract: The original version of this article unfortunately contained an error in two equations on page 25, inside the proof of Proposition 6.1. PubDate: 2019-09-01

Abstract: Abstract We consider well-known decomposition techniques for multistage stochastic programming and a new scheme based on normal solutions for stabilizing iterates during the solution process. The given algorithms combine ideas from finite perturbation of convex programs and level bundle methods to regularize the so-called forward step of these decomposition methods. Numerical experiments on a hydrothermal scheduling problem indicate that our algorithms are competitive with the state-of-the-art approaches such as multistage regularized decomposition, nested decomposition and stochastic dual dynamic programming. PubDate: 2019-09-01

Abstract: Abstract In this paper, we present a subspace method for solving large scale nonlinear equality constrained optimization problems. The proposed method is based on a SQP method combined with the limited-memory BFGS update formula. Each subproblem is solved in a theoretically suitable subspace. In the case of few constraints, we show that our search direction in the subspace is equivalent to that of the SQP subproblem in the full space. In the case of many constraints, we reduce the number of constraints in the subproblem and we show that the solution of the subspace subproblem is a descent direction of a particular exact penalty function. Global convergence properties of the proposed method are given for both cases. Numerical results are given to illustrate the soundness of the proposed model. PubDate: 2019-09-01

Abstract: Abstract Interior point methods (IPM) rely on the Newton method for solving systems of nonlinear equations. Solving the linear systems which arise from this approach is the most computationally expensive task of an interior point iteration. If, due to problem’s inner structure, there are special techniques for efficiently solving linear systems, IPMs demonstrate a reduced computing time and are able to solve large scale optimization problems. It is tempting to try to replace the Newton method by quasi-Newton methods. Quasi-Newton approaches to IPMs either are built to approximate the Lagrangian function for nonlinear programming problems or provide an inexpensive preconditioner. In this work we study the impact of using quasi-Newton methods applied directly to the nonlinear system of equations for general quadratic programming problems. The cost of each iteration can be compared to the cost of computing correctors in a usual interior point iteration. Numerical experiments show that the new approach is able to reduce the overall number of matrix factorizations. PubDate: 2019-09-01

Abstract: Abstract We propose a family of spectral gradient methods, whose stepsize is determined by a convex combination of the long Barzilai–Borwein (BB) stepsize and the short BB stepsize. Each member of the family is shown to share certain quasi-Newton property in the sense of least squares. The family also includes some other gradient methods as its special cases. We prove that the family of methods is R-superlinearly convergent for two-dimensional strictly convex quadratics. Moreover, the family is R-linearly convergent in the any-dimensional case. Numerical results of the family with different settings are presented, which demonstrate that the proposed family is promising. PubDate: 2019-09-01

Abstract: Abstract For control problems with control constraints, a local convergence rate is established for an hp-method based on collocation at the Radau quadrature points in each mesh interval of the discretization. If the continuous problem has a sufficiently smooth solution and the Hamiltonian satisfies a strong convexity condition, then the discrete problem possesses a local minimizer in a neighborhood of the continuous solution, and as either the number of collocation points or the number of mesh intervals increase, the discrete solution convergences to the continuous solution in the sup-norm. The convergence is exponentially fast with respect to the degree of the polynomials on each mesh interval, while the error is bounded by a polynomial in the mesh spacing. An advantage of the hp-scheme over global polynomials is that there is a convergence guarantee when the mesh is sufficiently small, while the convergence result for global polynomials requires that a norm of the linearized dynamics is sufficiently small. Numerical examples explore the convergence theory. PubDate: 2019-09-01

Abstract: Abstract In this paper, we consider the nonconvex quadratically constrained quadratic programming (QCQP) with one quadratic constraint. By employing the conjugate gradient method, an efficient algorithm is proposed to solve QCQP that exploits the sparsity of the involved matrices and solves the problem via solving a sequence of positive definite system of linear equations after identifying suitable generalized eigenvalues. Specifically, we analyze how to recognize hard case (case 2) in a preprocessing step, fixing an error in Sect. 2.2.2 of Pong and Wolkowicz (Comput Optim Appl 58(2):273–322, 2014) which studies the same problem with the two-sided constraint. Some numerical experiments are given to show the effectiveness of the proposed method and to compare it with some recent algorithms in the literature. PubDate: 2019-09-01

Abstract: Abstract We consider a family of dense initializations for limited-memory quasi-Newton methods. The proposed initialization exploits an eigendecomposition-based separation of the full space into two complementary subspaces, assigning a different initialization parameter to each subspace. This family of dense initializations is proposed in the context of a limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) trust-region method that makes use of a shape-changing norm to define each subproblem. As with L-BFGS methods that traditionally use diagonal initialization, the dense initialization and the sequence of generated quasi-Newton matrices are never explicitly formed. Numerical experiments on the CUTEst test set suggest that this initialization together with the shape-changing trust-region method outperforms other L-BFGS methods for solving general nonconvex unconstrained optimization problems. While this dense initialization is proposed in the context of a special trust-region method, it has broad applications for more general quasi-Newton trust-region and line search methods. In fact, this initialization is suitable for use with any quasi-Newton update that admits a compact representation and, in particular, any member of the Broyden class of updates. PubDate: 2019-09-01

Abstract: Abstract We propose a local regularization of elliptic optimal control problems which involves the nonconvex \(L^q\) quasi-norm penalization in the cost function. The proposed Huber type regularization allows us to formulate the PDE constrained optimization instance as a DC programming problem (difference of convex functions) that is useful to obtain necessary optimality conditions and tackle its numerical solution by applying the well known DC algorithm used in nonconvex optimization problems. By this procedure we approximate the original problem in terms of a consistent family of parameterized nonsmooth problems for which there are efficient numerical methods available. Finally, we present numerical experiments to illustrate our theory with different configurations associated to the parameters of the problem. PubDate: 2019-09-01

Abstract: Abstract Bounds are established for integration matrices that arise in the convergence analysis of discrete approximations to optimal control problems based on orthogonal collocation. Weighted Euclidean norm bounds are derived for both Gauss and Radau integration matrices; these weighted norm bounds yield sup-norm bounds in the error analysis. PubDate: 2019-09-01

Abstract: Abstract We revisit the classical Douglas–Rachford (DR) method for finding a zero of the sum of two maximal monotone operators. Since the practical performance of the DR method crucially depends on the step-sizes, we aim at developing an adaptive step-size rule. To that end, we take a closer look at a linear case of the problem and use our findings to develop a step-size strategy that eliminates the need for step-size tuning. We analyze a general non-stationary DR scheme and prove its convergence for a convergent sequence of step-sizes with summable increments in the case of maximally monotone operators. This, in turn, proves the convergence of the method with the new adaptive step-size rule. We also derive the related non-stationary alternating direction method of multipliers. We illustrate the efficiency of the proposed methods on several numerical examples. PubDate: 2019-09-01

Abstract: Abstract A strongly orthogonal decomposition of a tensor is a rank one tensor decomposition with the two component vectors in each mode of any two rank one tensors are either colinear or orthogonal. A strongly orthogonal decomposition with few number of rank one tensors is favorable in applications, which can be represented by a matrix-tensor multiplication with orthogonal factor matrices and a sparse tensor; and such a decomposition with the minimum number of rank one tensors is a strongly orthogonal rank decomposition. Any tensor has a strongly orthogonal rank decomposition. In this article, computing a strongly orthogonal rank decomposition is equivalently reformulated as solving an optimization problem. Different from the ill-posedness of the usual optimization reformulation for the tensor rank decomposition problem, the optimization reformulation of the strongly orthogonal rank decomposition of a tensor is well-posed. Each feasible solution of the optimization problem gives a strongly orthogonal decomposition of the tensor; and a global optimizer gives a strongly orthogonal rank decomposition, which is however difficult to compute. An inexact augmented Lagrangian method is proposed to solve the optimization problem. The augmented Lagrangian subproblem is solved by a proximal alternating minimization method, with the advantage that each subproblem has a closed formula solution and the factor matrices are kept orthogonal during the iteration. Thus, the algorithm always can return a feasible solution and thus a strongly orthogonal decomposition for any given tensor. Global convergence of this algorithm to a critical point is established without any further assumption. Extensive numerical experiments are conducted, and show that the proposed algorithm is quite promising in both efficiency and accuracy. PubDate: 2019-08-31

Abstract: Abstract In this paper is developed an accelerated version of the steepest descent method by a two-step iteration. The new algorithm uses information with delay to define the iterations. Specifically, in the first step, a prediction of the new test point is calculated by using the gradient method with the exact minimal gradient steplength and then, a correction is computed by a weighted sum between the prediction and the predecessor iterate of the current point. A convergence result is provided. In order to compare the efficiency and effectiveness of the proposal, with similar methods existing in the literature, numerical experiments are performed. The numerical comparison of the new algorithm with the classical conjugate gradient method shows that our method is a good alternative to solve large-scale problems. PubDate: 2019-08-28

Abstract: Many numerical methods for conic problems use the homogenous primal–dual embedding, which yields a primal–dual solution or a certificate establishing primal or dual infeasibility. Following Themelis and Patrinos (IEEE Trans Autom Control, 2019), we express the embedding as the problem of finding a zero of a mapping containing a skew-symmetric linear function and projections onto cones and their duals. We focus on the special case when this mapping is regular, i.e., differentiable with nonsingular derivative matrix, at a solution point. While this is not always the case, it is a very common occurrence in practice. In this paper we do not aim for new theorerical results. We rather propose a simple method that uses LSQR, a variant of conjugate gradients for least squares problems, and the derivative of the residual mapping to refine an approximate solution, i.e., to increase its accuracy. LSQR is a matrix-free method, i.e., requires only the evaluation of the derivative mapping and its adjoint, and so avoids forming or storing large matrices, which makes it efficient even for cone problems in which the data matrices are given and dense, and also allows the method to extend to cone programs in which the data are given as abstract linear operators. Numerical examples show that the method improves an approximate solution of a conic program, and often dramatically, at a computational cost that is typically small compared to the cost of obtaining the original approximate solution. For completeness we describe methods for computing the derivative of the projection onto the cones commonly used in practice: nonnegative, second-order, semidefinite, and exponential cones. The paper is accompanied by an open source implementation. PubDate: 2019-08-26

Abstract: Abstract The purpose of this paper is to study the problem of computing unitary eigenvalues (U-eigenvalues) of non-symmetric complex tensors. By means of symmetric embedding of complex tensors, the relationship between U-eigenpairs of a non-symmetric complex tensor and unitary symmetric eigenpairs (US-eigenpairs) of its symmetric embedding tensor is established. An Algorithm 3.1 is given to compute the U-eigenvalues of non-symmetric complex tensors by means of symmetric embedding. Another Algorithm 3.2, is proposed to directly compute the U-eigenvalues of non-symmetric complex tensors, without the aid of symmetric embedding. Finally, a tensor version of the well-known Gauss–Seidel method is developed. Efficiency of these three algorithms are compared by means of various numerical examples. These algorithms are applied to compute the geometric measure of entanglement of quantum multipartite non-symmetric pure states. PubDate: 2019-08-24

Abstract: Abstract We apply novel inner-iteration preconditioned Krylov subspace methods to the interior-point algorithm for linear programming (LP). Inner-iteration preconditioners recently proposed by Morikuni and Hayami enable us to overcome the severe ill-conditioning of linear equations solved in the final phase of interior-point iterations. The Krylov subspace methods do not suffer from rank-deficiency and therefore no preprocessing is necessary even if rows of the constraint matrix are not linearly independent. By means of these methods, a new interior-point recurrence is proposed in order to omit one matrix-vector product at each step. Extensive numerical experiments are conducted over diverse instances of 140 LP problems including the Netlib, QAPLIB, Mittelmann and Atomizer Basis Pursuit collections. The largest problem has 434,580 unknowns. It turns out that our implementation is more robust than the standard public domain solvers SeDuMi (Self-Dual Minimization), SDPT3 (Semidefinite Programming Toh-Todd-Tütüncü) and the LSMR iterative solver in PDCO (Primal-Dual Barrier Method for Convex Objectives) without increasing CPU time. The proposed interior-point method based on iterative solvers succeeds in solving a fairly large number of LP instances from benchmark libraries under the standard stopping criteria. The work also presents a fairly extensive benchmark test for several renowned solvers including direct and iterative solvers. PubDate: 2019-06-06