Do we have to swap at the same time?

No, not necessarily.

In fact, many of our members arrange non-simultaneous exchanges, where your guest may come to stay with you in June, but you don't stay at their's until December, as an example.