Before joining Swans, Westberg started a band with a friend from a punk rock club, calling themselves "Bookies", and was a member of Carnival Crash.

Westberg was for a short-time a member of The Heroine Sheiks with Shannon Selberg (ex-The Cows)[8] as well as John Fell (ex-China Shop).

Aside from the present incarnation of Swans he has been playing in the NYC bands NeVAh (with Vinnie Signorelli & Algis Kizys) and Five Dollar Priest (formerly Size Queens) (with Ron Ward of Speedball Baby & Bob Bert among others).

[9] Westberg also appeared (credited as Norman Westburg) in a short film The Right Side of My Brain (1985) directed by Richard Kern and starring Lydia Lunch.

Maple), Jim Sykes (Invisible Things), Matthew Wascovich (Scarcity Of Tanks), and Mike Watt (Minutemen) to form the art rock project band Hidden Rifles.

[15][4] Westberg draws inspiration from blues guitarists, which include Mick Ronson, Jimi Hendrix, Steve Hunter, Dick Wagner, J. Geils, Ritchie Blackmore, Tony Iommi, Ron Asheton, James Williamson, Andy Gill, Rowland S. Howard, Stevie Shears, Keith Levene, Tom Verlaine, and Richard Lloyd.