The test has low power (efficiency) for moderate to large sample sizes.
The relevant aspect of the median test is that it only considers the position of each observation relative to the overall median, whereas the Wilcoxon–Mann–Whitney test takes the ranks of each observation into account.
The test is sensitive also to differences in scale parameters and symmetry.
It is easy to prove by simulations, where samples with equal medians, yet different scales and shapes, lead the Wilcoxon–Mann–Whitney U test to fail completely.
Moreover, Siegel & Castellan (1988, p. 124) suggest that there is no alternative to the median test when one or more observations are "off the scale."