The worship itself is felt to be more important than the house in which it is to be celebrated." (Adeney)
"There cannot be a temple without an altar, but there may be an altar without a temple. God meets men at the place of sacrifice, even though there be no house for His name." (Maclaren)
During their stay in Babylon thye were not able to offer any sacrifices. That had to be done in the temple.