Linear inequalities via least squares

The Gaussian elimination method is usually used in solving problems related to systems of linear inequalities. The present review paper describes the application of the least-squares method to studying problems connected with linear inequalities (like redundant inequalities, theorems of alternative, mathematical programming). The minimum norm solution to the system of linear inequalities is found by solving a non-negative least-squares (NNLS) problem. A linear programming (LP) problem is transformed to the system of inequalities in several ways. By solving the corresponding NNLS problem an initial solution to the LP problem is found. The main ideas are explained by simple examples.


INTRODUCTION
Linear inequalities are more complicated than linear equations, because equations as constraints can be transformed into linear inequalities by replacing each equation with the opposing pair of inequalities.But there is no way a linear inequality can be transformed into equations.
The elimination method for solving linear equations remained unknown in Europe until K. Gauss rediscovered it in the 19th century.However, until recently there was no computationally viable method for solving systems of linear constraints including inequalities.Examples of linear constraints with inequalities started to appear in published literature in the mid-18th century.In the 19th and early 20th centuries J. Fourier, J. Farkas, L. Kantorovich and others initiated studies on solving such systems.The simplex method was invented by G. B. Dantzig in 1948 (see Dantzig and Thapa, 2003).
Today the applications of linear inequalities in almost all areas of science are so numerous, so well known and recognized that we do not have to repeat them here.Gale (1969) recommended to use for inequalities the lexicographic variant of the simplex method of Dantzig et al. (1955).
The least-squares method is described thoroughly in books by Lawson and Hanson (1995) and Björck (1996).Systems of linear inequalities are considered by Kuhn and Tucker (1956), Hu (1970), Chernikov (1971), and Papadimitriou and Steiglitz (1982).The present paper deals with the solution of a system of linear inequalities and other problems by applying the least-squares method (see Section 4).This highly developed method is much older than the simplex method.It is used not only in mathematics but also in statistics, physics, etc. Mainly nonlinear problems are solved by composing a certain number of similar linear least-squares problems, differing in a variable or constraint.In this paper it will be proved that such an idea can be used also for solving systems of linear inequalities and mathematical programming problems.
First of all, the least-squares method is recommended for degenerate and unstable problems.Orthogonal transformation matrices have a natural place in least-squares computations because they leave Euclidean length of a vector invariant.
The basic problem used in this paper is the non-negative least-squares (NNLS) problem, minimizing Ex − b 2 s.t.x ≥ 0, see Section 2. This problem is equivalent to phase I algorithm for the simplex method discussed by Leichner et al. (1993).Their algorithm solves least-squares subproblems and guarantees strict improvement of degenerate problems at each step.A similar algorithm based on the least-squares method was described by Übi (1989).
Problems related to redundant inequalities and theorems of alternative are considered in Section 3. In Section 4 a system of linear inequalities is solved using the least-squares method.Sections 5 and 6 present several cases of transforming linear and quadratic programming problems to systems of linear inequalities.For example, it is proposed to transform a multiobjective linear programming problem (LP problem) to the system of linear inequalities.
The exact methods for solving a system of linear inequalities can for the most part be categorized as either modified simplex-type methods or projection methods, which are based upon projections onto active sets of constraints.The projection methods are usually more efficient for the system of linear inequalities and related problems and require less storage than the modified simplex-type methods.In this paper we present a projection-type dual algorithm for solving a system of linear inequalities (Section 4).
The main goal of this article is to show how the least-squares method can be applied to studying a system of linear inequalities.

ALGORITHMS FOR THE NON-NEGATIVE LEAST-SQUARES (NNLS) PROBLEM
Let us have an NNLS problem, an overdetermined or underdetermined system of linear equations where Lawson and Hanson, 1995;Björck, 1996).In these books the main attention is paid to overdetermined systems, but in the mathematical programming, where m < n, working with systems of normal equations is more labour-consuming.
Assumption 2.1.The columns of the matrix E and vector b are unit vectors,

Seunghyun Kong's NNLS algorithm
Like the simplex algorithm, the NNLS algorithm by Kong (2007) has a basis B consisting of a set of linearly independent columns of E used for solving a system of equations.A feasible basis B is a set of linearly independent columns of E that have all positive components in the solution of where x * B is its solution.Other components x E\B = 0.Each major iteration of the NNLS solves with a feasible basis B. It requires at least one minor iteration.The minor iteration is a repeated process (inside a major iteration) of solving unconstrained minimization problems where B is the subset of B that remains after dropping some columns out of B, corresponding to non-positive components of x B .At each major iteration, we choose a column E s with (ρ, E s ) > 0, where the residual ρ = b − Bx B .The NNLS algorithm starts with an empty basis B, ρ = b.Since there is no zero-component solution of the least-squares problem, the solution from each major iteration improves strictly so that there is no degeneracy.The columns corresponding to zero and negative components in the solution are removed even though they are linearly independent of the other columns in B.
Algorithm NNLS: 4) is non-positive, then 5. solve problem (5) while all components x B corresponding to B are positive 6. end if; 9. end if; 10. end while; 11. x is optimal solution with min b − Ex E 2 > 0.

The algorithm by Dantzig and Thapa
The method by Dantzig and Thapa (2003) is similar to the previous one, where the initial basis for the simplex method is found.Start with the basis for At the start of an iteration a basis B is given, with the corresponding components then select an incoming column solving the least-squares problem Then, similarly to the previous algorithm, the inner cycle starts, during which a certain number of columns are eliminated from the basis B. The components of the vector x B corresponding to these columns are nonpositive.At the same time the sum of squares ρ 2 decreases at each step.

The algorithm by Übi
The third method for solving the problem (1) is described by Übi (2010).It corresponds to the first version of the second method (see Lawson and Hanson, 1995, ch. 24).For the matrix E the QR transformation is computed, where Q is an orthogonal and R is an upper triangular matrix.The order of the matrix R is one at the first iteration, two at the second iteration, etc. Choose the starting point x 0 = 0; the working set of columns is empty.At each step one variable x j0 is activated (for which column E j0 forms a minimal angle to the residual ρ = b − Ex E and column E j0 is added to the triangular matrix R) or one variable x j ≤ 0 (and its corresponding column in R) is removed.In the last case all the columns of R corresponding to this and following variables are replaced by the originals from the system (1).Then these Householder transformations which were performed before the variable x j (x j ≤ 0) are applied to the replaced columns of R. At last the replaced part is transformed into triangular form keeping the reflection normals in the subdiagonal part of R. The Householder transformations are memorized as products (see Lawson and Hanson, 1995).This textbook presents stability analysis of algorithms, where mn 2 − n 3 /3 operations are needed for solving the least-squares problem (1).

REDUNDANT INEQUALITIES AND THEOREMS OF ALTERNATIVE
Let us consider a system of linear inequalities where A, b, x are given matrices with dimensions m × n, m × 1, n × 1.
Assumption 3.1.The rows a i of the matrix A are non-zero vectors, a i = 0, i = 1, ..., m.
Definition 3.1.An inequality (a i , x) ≤ b i of the system Ax ≤ b is called redundant if (a i , x) ≤ b i is a valid inequality for the system obtained from Ax ≤ b by removing the inequality (a i , x) ≤ b i .
Each inequality can be tested for redundancy by solving a linear programming problem, but in certain cases only simpler rules can be applied (see Telgen, 1981;Cheng, 1987;Bazaraa et al., 2005).We shall study the application of the least-squares method to solve that problem.Definition 3.2.We say that a system of inequalities Ax ≤ b implies the inequality Let us multiply Ax ≤ b by y = (y 1 , ..., y m ) on the left and define It appears that the opposite statement is also true.
Let us compose an NNLS problem for the systems ( 7) and ( 8): This system has a non-negative solution if and only if Ax ≤ b implies (c, x) ≤ d.The least-squares method can be similarly used in the case of Theorems 3.1-3.5 (see Hu, 1970;Dantzig and Thapa, 2003).
Theorem of alternative 3.1.(Gale, 1969) The system Ax ≤ b has no solution if and only if there is a vector y such that

Khachiyan's algorithm
This algorithm decides whether the system of inequalities Ax ≤ b is consistent or not.Khachiyan (1980) assumed that the entries in A and b are all integers.Each step of Khachiyan's algorithm determines a point x k and ellipsoid At each step of the algorithm the associated ellipsoid contains a solution to the given system of linear inequalities, if one exists.The algorithm updates x k and Q k in a such way that the ellipsoid at the next step is smaller than that of the current step, but at the same time is guaranteed to contain a solution to the given system of inequalities (if one exists).However, Khachiyan's algorithm is of little practical value for solving real-world systems of inequalities and LP problems.For a detailed rigorous treatment of the method, we refer the reader to Bland et al. (1981) and Papadimitriou and Steiglitz (1982).

Gale's algorithm
A well-known approach for solving a system of linear inequalities Ax ≤ b is based on introducing artificial variables for the simplex method.Gale (1969) used a different method which is based on Theorem 3.1 presented in the previous section.He formulated an LP problem min{z}, If z min = 0, then the system has no solution.In the opposite case z min = 1 and the optimal solution to the dual problem Ax ≤ b is found from the optimal simplex table.

Least-squares method
Let us consider now finding a minimum norm solution to the system of linear inequalities Ax ≤ b Lawson and Hanson (1995) proved that the problem ( 11) is equivalent to the least-squares problem where u ∈ R m .Let us compose the dual problem for (11): y ≥ 0.
Equation ( 15) was proved by Lawson and Hanson (1995); equation ( 16) follows from the constraints of the dual problem (14) and equation (15).Vector x is the gradient of the objective function (11).It is expressed as a linear combination of the constraints gradients with non-positive coefficients, see equation ( 15).
The non-negative least-squares solution to the problem ( 12) is u = (0, 1/2), x = (0, 1) T .In this example only the variable u 2 and the respective constraints −x 2 ≤ −1 are active.The first constraint is also tight but inactive.The least-squares method uses only positive variables.
The decrease in the objective function ( 13) is guaranteed at each step.But the simplex method may stall in the degenerate case, performing a number of iterations at a degenerate vertex before producing any improvement in the objective value (see Leichner et al., 1993).
In Übi (2007) a minimum norm solution is found to the system of linear inequalities and the results of some test problems are given.

TRANSFORMING LP PROBLEMS TO THE SYSTEM OF INEQUALITIES
We consider the LP problem min{z where A, b, c are given matrices with dimensions m × n, m × 1, 1 × n, x and y are n × 1 and 1 × m vectors of variables.

The least-squares method
If primal and dual problems have solutions, then the optimal solutions of these problems satisfy the system of linear inequalities The solution of minimum norm to this system may be found by solving the respective NNLS problem (12).

Khachiyan's algorithm
Khachiyan (1980) proposed a polynomial-time algorithm for determining a solution to the LP problem (if one exists).It is based on the ellipsoid method which was originally developed by N. Shor, D. Yudin, and A. Nemirovski (see Papadimitriou and Steiglitz, 1982) for convex nonlinear programming.Khachiyan was the first to show that linear programming was in the class P of polynomial-time solvable problems.He reduced an LP problem to solving a certain number system of linear inequalities.The ellipsoid method can efficiently be used for convex programming and theoretically analysing combinatorial optimization problems, but the performance in practice for linear programming was very close to its worst-case bound.

The initial solution by shifting in the dual problem
Assumption 5.1.The right-hand side of the problem ( 19) is a positive vector: When the primal-dual simplex method is used, it is usually presumed that the feasible solution to the dual problem is known.If the feasible solution is not readily available, then a non-negative variable x n+1 should be introduced and an equation added to the constraints in ( 19), x 1 + x 2 + . . .+ x n+1 = b m+1 with b m+1 a very large positive constant.Then the feasible solution to the dual problem is available with y 1 = y 2 = . . .= y m = 0, y m+1 = min j c j .However, such an initial solution is usually very far from being an optimal one.As the initial solution is easy to find, this problem has earlier not been studied in detail.Übi (2005, 2010) make a coordinate shift in the dual problem, enabling finding via the least-squares method the initial solution which may turn out to be optimal.
Let us make a shift of coordinates In order to solve this problem, Harris (1975) changes the norms of columns and as a result the number of steps of the simplex method decreases.But in the case of the shift t = 10, the new origin of coordinates is O'(20,10,20/3) and the right-hand side c = (16/3, −276/3, −359/3, −18, −8, −14/3).By solving the respective NNLS problem (12) we find the minimum norm solution y = (−18, −25/3, −14/3) to the system y A ≤ c and the corresponding solution to the initial dual problem y * = y + 10b = (2, 5/3, 2).The last solution is optimal.
If the objective function of the dual problem is bounded and the shift sufficiently large, then the optimal solution to the dual problem is a feasible solution of minimum norm (see Übi, 2005).However, a scaling problem arises in the respective NNLS problem when t is too big.To avoid this, Übi (2007) recommended to use a "moderately large" shift to find the initial solution to the primal-dual simplex method.
Remark 5.1.In addition to yielding a better initial solution, the least-squares method is effective also in case of a degenerate basis.Strict improvement of the objective function is attained at each iteration, see Leichner et al. (1993).

The initial solution for the chosen value of the objective function
N. Karmarkar (see Papadimitriou and Steiglitz, 1982) assumed that the minimum of the objective function for the LP problem is known.Übi (2010) assumed only the knowledge of the estimate on this value z 0 .The parameter z 0 can be estimated more easily if the problem (19) contains the constraints lower ≤ x ≤ upper.
Let us write for the dual problem a system of linear inequalities which in case of the parameter z 0 = z min = w max has unique minimum norm solution ŷ, which in a general case is one of the optimal solutions to the dual problem.Let us formulate the NNLS problem (12) corresponding to this system of inequalities: In Übi (2010) the proofs of the following two theorems are given.
Remark 5.2.If the LP problem ( 19) is infeasible, then for every z 0 the system of linear inequalities ( 22) holds.We solve the problem ( 23) and obtain by equation ( 15) a feasible solution to the dual problem (20).
Assumption 5.3.Let us assume that the primal problem ( 19) and the dual problem (20) have optimal solutions.
Case I: For the chosen parameter z 0 the inequality holds.
Theorem 5.2.In case of inequality (24), the problem (23) has an exact solution u 0 , u i , with the corresponding x, Case II: Finding the least-squares solution to the problem (23), with the help of equation ( 15) we can calculate a feasible solution to the dual problem (20) with the value of the objective function not less than z 0 .
In linear programming it is often sufficient to have a solution for which the objective function of the initial problem is sufficiently small or the objective function of the dual problem is sufficiently large.For example, for an appropriate w 0 let us solve the problem (y, b) ≥ w 0 , yA ≤ c using the least-squares method.The same approach can be applied in goal programming, where each objective function can be replaced by a constraint.
If the LP problem is degenerate, ill-conditioned or we have any initial basis, then it is expedient to start the solution process with the least-squares problem (23).

SOLVING QUADRATIC PROGRAMMING PROBLEMS
Hildreth (1957) proposed a quadratic programming algorithm which solves the dual problem -minimizing a quadratic function in the positive orthant.In this section we describe two methods for a strictly convex quadratic programming problem which are based on Hildreth's and least-squares methods.

Iterative row-action method
Let us have a quadratic programming problem min{0.5(y,By) where where A = GD −1 and b = h + GB −1 d.Lent and Censor (1980) presented an iterative relaxation procedure for the problem ( 27) which is an extension of Hildreth's quadratic programming algorithm, specially designed for approximating the minimum-norm element of a polyhedral convex set.They discussed also the option of the relaxation parameter.The presented algorithm is suitable for handling large (or huge) and sparse problems, because no changes in the original matrix are made.At each iteration access is required to only one row of the matrix (see Censor and Zenias, 1997).

The least-squares method
The problem ( 26) is transformed to the NNLS problem (12) and solved using the algorithm QPLS.The QR transformation is computed and orthogonal transformations are memorized as products.Assuming g j ≤ 0, j = 1, . . ., m, the equality constraints Cx = g are written in the form Cx ≤ g.The numerical testing on some difficult problems presented in Übi (2008) indicate that the algorithm QPLS gives excellent accuracy.

CONCLUSION
The highly developed least-squares technique is used in many areas of science.That method can be applied to a system of linear inequalities which is transformed to the non-negative least-squares (NNLS) problem.Three finite algorithms for solving the NNLS problem are described, based on QR decomposition of the coefficients matrix.The NNLS problem can be formulated also for large problems, because the matrix A of the system Ax ≤ b is not transformed.
Redundant inequalities, theorems of alternative, and linear and quadratic programming problems are studied using the NNLS problem.The numerical testing on some difficult problems indicates that the leastsquares method gives excellent accuracy.
The disadvantage of the proposed least-squares method for solving systems of linear inequalities is that it does not treat simple bounds on the variables directly.
Using the Choleski decomposition with B = D T D and shifting y = D −1 x − B −1 d, the problem (26) is transformed to the problem (11) -finding the solution of minimum norm to the system of linear inequalities min{z = 0.5 x 2 }, Ax ≤ b,

Evald
Übi received his PhD in mathematical cybernetics from Kiev Institute of Cybernetics of Ukrainian Academy of Sciences in 1978.Currently he is Associate Professor at the Chair of MathematicalEconomics at Tallinn University of Technology, Estonia.His main research interests are system of linear inequalities, linear, quadratic, and stochastic programming, method of least squares.