In the context of the film it acts as a kind of pun in relation to Harry's double bass looking like an oversized fiddle.
The plot paralleled the Harold Lloyd film The Kid Brother released earlier that year.
Harry does not realise his own shortcomings as a music student, whose instrument of choice is the double bass.
At first he joins an international street band, headed by Prof von Tempo - but they decide he is not what they want.
The junkman realises the potential of stuff thrown out of windows at Harry (to make him shut up) as this can be collected and sold.
Next Harry drives up in a huge steam roller and accidentally crushes the piano.