[Nyororin]

It may not make a difference... If you are writing very slow, very carefully, and all while paying attention to balance and the like.
But if you write any faster, it will both become obvious that you are writing them in the incorrect order and possibly even become illegible.

With messier handwriting or quick writing that borders on "script" - it matters immensely. If you`re writing with the correct stroke order your pen can never leave the paper and still be totally legible, as the stroke order will leave a distinct pattern.

It`s really hard to explain, but does indeed make a huge difference in legibility. In some cases, with certain hiragana or katakana it may not matter (も being one that drives both me and my son crazy as it seems to defy the normal rule of stroke order... Plus everyone in the family writes it the "wrong" way)- but kanji is pretty unforgiving.