One motivation was to construct moduli spaces in algebraic geometry as quotients of schemes parametrizing marked objects.
The first one, successfully tackled by Hilbert in the case of a general linear group, is to prove that the algebra A is finitely generated.
Whether a similar fact holds for arbitrary groups G was the subject of Hilbert's fourteenth problem, and Nagata demonstrated that the answer was negative in general.
The finite generation of the algebra A is but the first step towards the complete description of A, and progress in resolving this more delicate question was rather modest.
Rather remarkably, unlike his earlier work in invariant theory, which led to the rapid development of abstract algebra, this result of Hilbert remained little known and little used for the next 70 years.
(The book was greatly expanded in two later editions, with extra appendices by Fogarty and Mumford, and a chapter on symplectic quotients by Kirwan.)
i.e. the quotient space of X by the group action, runs into difficulties in algebraic geometry, for reasons that are explicable in abstract terms.
As Mumford put it in the Preface to the book:The problem is, within the set of all models of the resulting birational class, there is one model whose geometric points classify the set of orbits in some action, or the set of algebraic objects in some moduli problem.In Chapter 5 he isolates further the specific technical problem addressed, in a moduli problem of quite classical type — classify the big 'set' of all algebraic varieties subject only to being non-singular (and a requisite condition on polarization).
As Mumford puts it, if the first two difficulties are resolved [the third question] becomes essentially equivalent to the question of whether an orbit space of some locally closed subset of the Hilbert or Chow schemes by the projective group exists.To deal with this he introduced a notion (in fact three) of stability.
This enabled him to open up the previously treacherous area — much had been written, in particular by Francesco Severi, but the methods of the literature had limitations.
The concept was not entirely new, since certain aspects of it were to be found in David Hilbert's final ideas on invariant theory, before he moved on to other fields.