Ah ha, but some quests state HAVE 2 X's or 1 X, rather than build X...which means you actually have to build it.
As I have have just discovered! (unfortunately forgot the tavern and started the builds! Doh)
Now that's when a single 1 building speed up, in the inventory, would come in handy!