You can sort the items in your playlist by pressing the "Sort -> Sort by path and filename" button.
If you are using the new playlist style, then you have to rightclick in the playlist window and choose "Sort by path and filename" and all the files will be sorted as like in Windows.
If they are already sorted in the playlist, then probably you have to "Random" option enabled for the repeat mode.
To change it, use the default skin (some skins doesn't have the ability to change that value), click on the VIDEO button on the right of the controls window, and click on the circular arrow at the bottom right corner. Thake a look at the skin strings and stop to click that button one you reached the "Repeat off" setting (or the one you want to use)