[kwlug-disc] fun: 10k to a million threads and more?

[Erik Schnetter]

When people run millions of threads on a laptop, they usually use a
user-mode threading system, i.e. they don't use kernel (POSIX)
threads. User-mode threads consume fewer resources than kernel threads
(they have to save less state), and they can also be optimized for
different uses. For example, I would assume that kernel threads are
optimized for latency, whereas a user-mode threading system could be
optimized for throughput, so that letting a runnable thread wait for a
minute or so would be just fine. (That would be unacceptable for a
kernel thread where a person might be sitting in front of a keyboard,
pushing a key, and expecting it to show up on the screen.)

I have used HPX <https://hpx.stellar-group.org/> and Qthreads
<https://github.com/Qthreads/qthreads> in the past. (The QT threading
system also uses the term "qthread", but that's something completely
different.) My background here is in high-performance computing, and
depending on your use case that might or might not be close to what
you're looking for.

-erik

On Tue, May 30, 2023 at 4:40 AM Chris Frey <cdfrey at foursquare.net> wrote:
>
> Just out of curiosity, I'm trying to tweak my system to allow for the
> creation of a million idle threads.
>
> So far I can't even crack 10k. :-)
>
> I've attached a simple C++ program I'm using to test with.
>
> So far my tweaks have looked like this:
>
>         echo 200000 > /proc/sys/kernel/threads-max
>         echo 200000 > /proc/sys/kernel/pid_max
>         echo 600000 > /proc/sys/vm/max_map_count
>         ulimit -s 256
>         ulimit -i 200000
>         ulimit -n 1048576
>
> When I run maxthreads 9300 or so, lsof shows about 400k open files.
> Without maxthreads running, lsof shows about 30k.
>
> Any clues for more tweaks?
>
> Thanks,
> - Chris
>
> _______________________________________________
> kwlug-disc mailing list
> To unsubscribe, send an email to kwlug-disc-leave at kwlug.org
> with the subject "unsubscribe", or email
> kwlug-disc-owner at kwlug.org to contact a human being.



-- 
Erik Schnetter <schnetter at gmail.com>
http://www.perimeterinstitute.ca/personal/eschnetter/