[4] On October 14, 2024, Cain announced the release of Perverts and its lead single "Punish", after teasing visuals on social media with grainy and black-and-white pictures.

[25][20] The track also incorporates the sound of a rusted swing set recorded at midnight,[26] which Clare Martin of Paste considered as a reference to childhood.

Pitchfork's Meaghan Garvey saw it reminiscent to Ruins (2014) by the musician Grouper, as well as the works of the singer-songwriter Midwife, who describes her music as "heaven metal".

[8] The Independent's Helen Brown compared it to the late-1990s ballads by the singer-songwriter Sarah McLachlan,[27] and The Fader's David Renshaw interpreted it as a mix between Grouper and the singer Lana Del Rey.

[17] Additionally, Sanders drew similarities between the piano progressions of "Punish" and "Horizon" (2017), a song by the indie folk singer-songwriter Aldous Harding.

[31] In a press release regarding the song, Cain stated: "I wonder how deep shame can run, and how unforgivable an act could be that I may still justify it in some bent way to make carrying it more bearable.

[1] Writing for Clash, Robin Murray opined that the song "illustrates how gripping, and how emotionally redolent her work can be", and that it "acts as musical therapy, stripping away the layers of dirt on her life".

[28] Sasha Geffen of Pitchfork praised Cain's vocal performance on the track as an improvement from her previous releases,[34] and Grant Sharples of Uproxx believed that she expressed the same sincerity and "deep conviction in herself and her art" as she did on Preacher's Daughter.

[a] "Punish" was ranked at number 19 on Dazed's listicle of the best tracks released in 2024; Serena Smith called the song "bleak but beautiful".

[40] The Fader's Sandra Song described the performance as exquisite, slow-moving, and "full of pensive melancholy",[41] while Stereogum's Tom Breihan said it was "long and meditative".