Although the Y claims that 20,000 square feet are unusable, its downtown location currently houses two pools (one with two water slides and a one meter diving board), two gyms, racquetball, handball and squash courts, a weight room, a cardio center and a gymnastics center, as well as administrative offices.

According to recent Y statements, the downtown facility has been inadequate for forty-eight years! Nevertheless, until the last decade, no serious public plans to move were floated. That changed in the late 1990s, when the Town of Westport purchased Baron's South, a 23 acre property at the corner of the Post Road and Compo Road South, for $7 million. The Y then proposed building a joint facility with the Senior Center on Baron's South, and even hired a noted architect to design a new 90,000+ square foot building.

At the time, the Y stated that it had considered alternative sites for a new building--including Camp Mahackeno, which the Y had owned since 1945--but that these sites were unsuitable for its needs. In addition, the Y argued that placing its new facility at Baron's South would enrich Westport's center. Despite an intense publicity campaign by the Y, the notion of a joint facility was ultimately rejected by the town, reportedly because the Y was willing to pay only $1.5 million for the 8 acres that it wanted for its building and parking lots.

Since its failure to secure Baron's South, the Y has continued a very public effort to look for alternative building sites, and reportedly has considered 37 possible locations in Westport and Weston. The Y rejected every one of them as too small, or too expensive, or too far from town center, or otherwise "unsuitable."

Although the Y has not made public either the locations it considered, or what specifications led to their rejection, some of the rejected locations reportedly included Winslow Park (town owned), the Imperial Avenue parking lot (also town owned), the former police barracks on the Post Road (now a mall-style drug store) and several parcels on Post Road West (one of which is now Terra Nova).

The Y also rejected suggestions that it expand its downtown facility by taking over either the town owned parking lot behind the Bedford Building or the Victorian office building next door on Church Street. Finally, the Y rejected the idea of building a combined facility with the Norwalk YMCA (which, ironically, was also funded by the Westport Y's benefactor, Edward Bedford).

In March 2004, the Y announced its intention to sell its historic home, leave downtown and build a new 94,000 square foot facility at Camp Mahackeno in remote, rural and residential northwestern Westport.