The most frequent use of the term modular equation is in relation to the moduli problem for elliptic curves.
That implies that any two rational functions F and G, in the function field of the modular curve, will satisfy a modular equation P(F,G) = 0 with P a non-zero polynomial of two variables over the complex numbers.
For suitable non-degenerate choice of F and G, the equation P(X,Y) = 0 will actually define the modular curve.
This can be qualified by saying that P, in the worst case, will be of high degree and the plane curve it defines will have singular points; and the coefficients of P may be very large numbers.
Such equations first arose in the theory of multiplication of elliptic functions (geometrically, the n2-fold covering map from a 2-torus to itself given by the mapping x → n·x on the underlying group) expressed in terms of complex analysis.