To follow up on this, after making the below changes while my main disk IO went down, my load average went up, memory usage went through the roof & swapping ensued. I increased the amount of memory assigned to each of my main nodes (those that gossip with the outside world) and it seems to be holding steady so far: https://imgur.com/a/b0S4Ui2
I believe the nodes may have also been having issues gossiping as I saw outbound network traffic flatline during the same time periods: https://imgur.com/a/IEoLboM
-T
Signed PGP part I also applied these configuration options earlier today to all the servers in 1 of my pools that was experiencing high IO load and repeated SigAlarms: command_timeout: 600
wserver_timeout: 30
max_recover: 150
And since then, everything has been quiet:
Hopefully this can help other operators keep things in better shape for the time being.
-T
With your suggestions: load average below 1 Traffic: ~150G/day Best, Rolf Am 2019-02-04 12:52, schrieb Martin Dobrev: Hi,
I've spent last week trying to optimize configuration as much as
possible. Following advise from a previous mail I've added:
command_timeout: 600
wserver_timeout: 30
max_recover: 150
to my sksconf and it seems this fixed majority of the EventLoop
failures. I've added DB_CONFIG in KDB/PTree folders to get rid of DB
archive logs that were causing plenty of IO load too.
My clusters are now happily responding to queries and load-average is
bellow one. Traffic wise things look better too, ~20GB/day.
Kind regards,
Martin Dobrev
P.S. Adding/changing DB_CONFIG might cause an error in the databases
that you can easily fix by running
db_recover -e -v -h <path to SKS>/{KDB,PTree}
On 04/02/2019 09:49, Rolf Wuerdemann wrote:
Hi,
Don't get me wrong, but within three days I've got 450G traffic
which can be assigned to sks by 99.9%. Estimated to 30 days this
means 4.5T (which is in good agreement of your 2+T/Key for these
two poison keys).
With this amount of traffic and the possibility to get
more of this keys (thus more traffic) every moment, I think it's
only a question of time until the network with the current
implementation will vanish. Traffic increased roughly a factor of
300 (15G->4.5T) within twelve months, nodes within the network
decreased by a factor of two at least for the same time.
So: where to go and how?
Just my 2ct,
rowue
Am 2019-01-30 22:09, schrieb Martin Dobrev:
Hi,
My observations so far show that both keys generate 2+ TB/month
traffic on average for all my clustered nodes. I'm running nginx +
Varnish in-memory cache tuned at 5 minutes TTL which gives plenty of
CPU cycles for the never-ending EventLoop alarm loops. The latter
cause load-average spikes of up to 10 with just 4 Docker containers
running on a 12 core system.
Don't get me wrong. The throttling penalty is something I'd
swallow-up
as long as we keep the network running.
Regards,
Martin
keyserver.dobrev.eu | pgp.dobrev.it
-------- Original message --------
From: Kristian Fiskerstrand
<address@hidden>
Date: 30/01/2019 20:18 (GMT+00:00)
To: Shengjing Zhu <address@hidden>, address@hidden
Subject: Re: [Sks-devel] Unusual traffic for key 0x69D2EAD9 and
0xB33B4659
On 1/12/19 8:15 PM, Shengjing Zhu wrote:
I think these requests are quite unusual.
Does anyone know what happens to these two keys?
Just to add a comment on this, adding a cache on the load-balancer
is
really a nice way to slow down hits on the underlying SKS nodes, I
keep
cache for 10 minutes in nginx, which really makes life more
pleasant.
--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Action is the foundational key to all success"
(Pablo Picasso)
_______________________________________________
Sks-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/sks-devel
_______________________________________________
Sks-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/sks-devel
-- Security is an illusion - Datasecurity twice Rolf Würdemann - address@hidden - DL9ROW GnuPG fingerprint: EEDC BEA9 EFEA 54A9 E1A9 2D54 69CC 9F31 6C64 206A xmpp: address@hidden E1189573 6B4A150C A0C2BF5A 5553F865 0B9CBF7A address@hidden 64CBBB68 0A3514A4 026FC1E7 5328CE87 AEE2185F _______________________________________________ Sks-devel mailing list address@hiddenhttps://lists.nongnu.org/mailman/listinfo/sks-devel
|
signature.asc