It features Suzanne Vega on background vocals, as well as guest musicians Frank Christian on acoustic guitar and Jeffrey Berman on vibraphone.

The song takes its title from the 1950 film noir In a Lonely Place, starring Humphrey Bogart and Gloria Grahame, and its pre-chorus lyrics are largely adapted from key lines of dialogue in the film; at one point, Bogart's character says, "I was born when she kissed me.

I lived a few weeks while she loved me," while in the film's climactic scene, Grahame's character says, "Yesterday, this would've meant so much to us.

The relevant lyrics are: "I was born the day I met you/Lived a while when you loved me/Died a little when we broke apart//Yesterday, it would have mattered/Now today, it doesn't mean a thing/All my hopes and dreams are shattered now."

Instead, DiNizio decided to record the song with Suzanne Vega, whom he knew from when they worked together at an office.