You expected much, but see, it turned out to be little. What you brought home, I blew away. Why?” declares the Lord Almighty. “Because of my house, which remains a ruin, while each of you is busy with his own house. The Bible, Haggai 1:9
When the Israelites came back from a multiple generation exile, they found Jerusalem in complete ruins. So they thought it would be good to rebuild the temple first.