Exactly that.On some 4t's,washing the bike without one can put water down your cylinder.On 2t's,at the very least you can destroy your packing-or if you washed it and left it sit long enough,rust out or weaken your pipe.
On that same note-cover your air filter with a plastic bag or something before washing as well.