Different species may employ different methods, but all of them are based on one or more senses (after all, this is how the organism gathers information about the environment).
The recognition may happen by chemical signature (smell), by having a distinctive shape or color (sight), by emitting certain sounds (hearing), or even by behaviour patterns.
Among human beings, the sense of sight is usually in charge of recognizing other members of the same species, with maybe the subconscious help of smell.
For example, ornithologists have great difficulty in distinguishing the chiffchaff from the willow warbler by eye, and there is no evidence that the birds themselves can do so other than by the different songs of the male.
[1] They have also been known to emit chemical cues in order to recognize members of their own species.