The office is not the United Kingdom's finance minister; this role is instead held by the Chancellor of the Exchequer, who is the second lord of the Treasury.
As of the beginning of the 17th century, the running of the Treasury was frequently entrusted to a commission, rather than to a single individual.
Eventually in the middle of the same century, the first lord of the Treasury came to be seen as the natural head of the overall ministry running the country, and, as of the time of Robert Walpole (Whig), began to be known, unofficially, as the prime minister.
By convention, the other Lords Commissioners of the Treasury are also Government Whips in the House of Commons.
10 Downing Street is the official residence of the first lord of the Treasury, not the office of prime minister.