Fix merge to support shuffling

[tkram01]

@lemire I've created a class called ExternalShuffle and changed the references from sort to shuffle. I removed the parallel option since there isn't a streams shuffle. I replaced everywhere a Comparator was being passed around to an instance of Random.

[lemire]

Got it. Give me some time to get back to it (it is late in Montreal right now).

[lemire]

It is unlikely that you genuinely want to use Random. Maybe ThreadLocalRandom?
Please see https://lemire.me/blog/2016/02/01/default-random-number-generators-are-slow/

Generally, assuming that we do not care about duplicates, I think
we should never, ever, need to compare two lines. So I would
expect to find no comparator in this code except if we care
about duplicates somehow... but then we would need to explain how
we handle them.

Let us look at shuffInBatch first. This repeatedly calls
shuffleAndSave which in turn calls Collections.shuffle(tmplist, random).
That looks reasonable to me. (I did not verify it carefully, but the
idea must be right.)

Next, let us look at mergeShuffledFiles. Ok. Here is a simple but
hopefully correct algorithm. What you do is to repeatedly pick a line
from one of the buffers and append it to the answer. However, you do not
need nor want a priority queue. Suppose that at time t, there remains
N lines to merge. And the buffers have N1, N2... N9 (assume 9 buffers)
lines each. We have that N1 + N2+ ... + N9 = N.
You pick a number x between 0 and N. If x < N1, pick the top line from
the first buffer and decrement N1. If x < N1+N2, pick the top line from
the second buffer and decrement N2, and so forth.

If you implement that, it is probably good already... benchmark it on large
files and see how well it fares.

The problem with this simple algorithm is that if the number of buffers is
large, then you have to potentially iterate over a large number each time.
That is not nice.

You can do better however if the numbers of buffers is large, and it is not
hard. So divide your buffers into two sets of buffers of size M1 and M2 so
that M1+M2 = N. Pick an integer in [0,N). If it is smaller than M1, decrement
M1 and pick a buffer in the corresponding set. If it is larger than or equal
to M1, then go with the other set of buffers (and decrement M2).

So we have reduced the size of the problem in two. We can repeat the same
trick recursively. We can discuss it later.

[lemire]

See https://lemire.me/blog/2020/04/26/sampling-efficiently-from-groups/