The architectural features of the Gaetani Tower House have allowed scholars to date its construction to the latter half of the 12th century. The original structure started to be altered as early as the 14th century, when the wooden balconies were torn down and the spaces between the pilasters were blocked with bricks and then pierced with mullioned windows.
It is believed that the edifice was erected in this area in relation to the construction of a second crossing over the River Arno (called Ponte Nuovo, Italian for ''new bridge''). Starting from 1241 the tower was called ''Torre De Cantone'' on account of its location on the corner (''canto'' or ''cantone'' being the Italian for ''corner'') of the Church of San Nicola.
The building had previously been known as ''Domus'' of the Dodi, and inside it there was an ''apotheka'', or shop. It has been shown that, on top of this tower, Galileo Galilei invited the grand duke to observe his first experiments with the telescope. It was not until the 19th century that the tower came to be known as ''Torre della Verga d''Oro'' (Tower of the Gold Rod), a name derived from that of a no longer extant structure standing on the south-eastern corner of the Via Santa Maria.