Re: [Sks-devel] SKS scaling configuration

[Jeremy T. Bouse]

Hi Todd,

    The timing of this thread and your reply are ideal as I'm in the process of working to fix my cluster that has been down for some time due to system failure and lack of available time on my part to repair it. Since I'm in the process I've been working to revisit the setup itself.

    I didn't have as many locations configured as you show in your example but it looked like you were defining the map but I didn't see it being used in any of your location blocks unless I'm missing something. Shouldn't you be using $upstream_server in your proxy_pass configuration?

    Having only one of the nodes gossiping outside the cluster definitely simplifies things compared to how I had things setup before, especially when moving the servers behind a firewall.

On 2/17/2019 12:18 PM, Todd Fleisher wrote:

Hi Jonathon,

I've previously spoken with Kristian about this off-list in an attempt to improve the performance & resilience of my own server(s) pool(s), so let me share his recommendations which I’ve been using with minimal issues.

The setup uses a caching NGINX server to reduce load on the backend nodes running SKS. His recommendation is to run at least 3 SKS instances in the backend (I’m running 4). Only one of the backend SKS nodes is configured to gossip with the outside world on the WAN, along with the other backend SKS nodes on the LAN. The NGINX proxy is configured to prefer that node (the one gossiping with the outside world - let’s call it the "primary") for stats requests with a much higher weight. As a quick aside, I’ve observed issues in my setup where the stats requests are often directed to the other, internal SKS backend nodes - presumably due to the the primary node timing out due to higher load when gossiping. This then gets cached by the NGINX proxy and continues to get served so my stats page reports only the internal gossip peer’s IP address vs. all of my external peers. If Kristian or anyone else has ideas on how to mitigate/minimize this, please do share. Whenever I check his SKS node @ http://keys2.kfwebs.net:11371/pks/lookup?op=stats I always find it reporting his primary node eta_sks1 with external & internal peers listed.

Here are the relevant NGINX configuration options. Obviously you need to change the server IP addresses & the hostname returned in the headers:

upstream sks_servers
{
       server 192.168.0.55:11372 weight=5;
       server 192.168.0.61:11371 weight=10;
       server 192.168.0.36:11371 weight=10;
}

upstream sks_servers_primary
{
       server 192.168.0.55:11372 weight=9999;
       server 192.168.0.61:11371 weight=1;
       server 192.168.0.36:11371 weight=1;
}

map $arg_op $upstream_server {
       "stats" sks_servers_primary;
       default sks_servers;
}

And in server context:

proxy_buffering on;
proxy_buffer_size 1k;
proxy_buffers 24 4k;
proxy_busy_buffers_size 8k;
proxy_max_temp_file_size 2048m;
proxy_temp_file_write_size 32k;

 location /pks/hashquery {
               proxy_pass http://sks_servers/pks/hashquery;
               proxy_pass_header  Server;
               add_header Via "1.1 keys2.kfwebs.net";
               proxy_ignore_client_abort on;
       }

       location /pks/add {
               client_max_body_size 5m;
               proxy_pass http://sks_servers/pks/add;
               proxy_pass_header  Server;
               add_header Via "1.1 keys2.kfwebs.net";
               proxy_ignore_client_abort on;
       }

       location /pks {
   proxy_cache backcache;
       proxy_ignore_headers Cache-Control "Expires";
   proxy_cache_valid any 10m;
  # proxy_cache_bypass $http_cache_control;
   add_header X-Proxy-Cache $upstream_cache_status;
               proxy_pass http://sks_servers/pks;
               proxy_pass_header  Server;
               add_header Via "1.1 keys2.kfwebs.net";
               proxy_ignore_client_abort on;
       }

Here is my sksconf file:

#  sksconf -- SKS main configuration

#

#basedir: /etc/sks

# debuglevel 3 is default (max. debuglevel is 10)

debuglevel: 3

hostname: sks.pod01.fleetstreetops.com

nodename: sks03.pod01

hkp_address: 127.0.0.1

hkp_port: 11371

recon_port: 11370

#

server_contact: 0xD16C3A41949D203A

#from_addr: address@hidden

#sendmail_cmd: /usr/sbin/sendmail -t -oi

#

initial_stat:

membership_reload_interval: 1

stat_hour: 17

#

# set DB file pagesize as recommended by db_tuner

# pagesize is (n * 512) bytes

# NOTE: These must be set _BEFORE_ [fast]build & pbuild and remain set

# for the life of the database files. To change a value requires recreating

# the database from a dump

#

# KDB/key 65536

pagesize: 128

#

# KDB/keyid 32768

keyid_pagesize: 64

#

# KDB/meta 512

meta_pagesize: 1

# KDB/subkeyid 65536

subkeyid_pagesize: 128

#

# KDB/time 65536

time_pagesize: 128

#

# KDB/tqueue 512

tqueue_pagesize: 1

#

# KDB/word - db_tuner suggests 512 bytes. This locked the build process

# Better to use a default of 8 (4096 bytes) for now

#word_pagesize: 8

#

# PTree/ptree 4096

ptree_pagesize: 8

disable_mailsync:

# Adding to try and address poison keys

command_timeout: 600

wserver_timeout: 30

max_recover: 150

As for my membership file(s), I try to keep all of them identical, just commenting out the various peers that are not needed outside of the primary node. So my primary node (10.x.x.207) would look like this:

10.x.x.226 11370

10.x.x.217 11370

#10.x.x.207 11370

10.x.x.221 11370

#10.y.x.30 11370

#10.y.x.31 11370

#10.y.x.32 11370

#10.y.x.26 11370

sks.pod02.fleetstreetops.com 11370

pgp.librelabucm.org 11370 # LibreLabUCM address@hidden 0x6FC10EAE0B5C3FC4

agora.cenditel.gob.ve 11370 # Lully Troconis <address@hidden> 0x4758944f58aad9e1

pgpkeys.co.uk 11370 # Daniel Austin <address@hidden> 0x34A3662F837F2C28

fks.pgpkeys.eu 11370 # Daniel Austin <address@hidden> 0x34A3662F837F2C28

sks.infcs.de 11370 # Steffen Kaiser <address@hidden> 5119CB3603B258AAC1EBA7A723A371DE9ABC764F

pgpkeys.urown.net 11370 # <address@hidden> 0x27A69FC9A1744242

While one of my non-primary nodes (10.x.x.226) would look like this:

#10.101.7.226 11370

#10.101.7.217 11370

10.101.7.207 11370

#10.101.7.221 11370

#10.102.10.30 11370

#10.102.10.31 11370

#10.102.10.32 11370

#10.102.10.26 11370

#sks.pod02.fleetstreetops.com 11370

#pgp.librelabucm.org 11370 # LibreLabUCM address@hidden 0x6FC10EAE0B5C3FC4

#agora.cenditel.gob.ve 11370 # Lully Troconis <address@hidden> 0x4758944f58aad9e1

#pgpkeys.co.uk 11370 # Daniel Austin <address@hidden> 0x34A3662F837F2C28

#fks.pgpkeys.eu 11370 # Daniel Austin <address@hidden> 0x34A3662F837F2C28

#sks.infcs.de 11370 # Steffen Kaiser <address@hidden> 5119CB3603B258AAC1EBA7A723A371DE9ABC764F

#pgpkeys.urown.net 11370 # <address@hidden> 0x27A69FC9A1744242

Another important item that was recently discussed on the list is to ensure you build your SKS DB with a DB_CONFIG file in your DB & PTree sub-directories to avoid issues with DB logfiles building up:

set_mp_mmapsize         268435456

set_cachesize 0 134217728 1

set_flags DB_LOG_AUTOREMOVE

set_lg_regionmax 1048576

set_lg_max 104857600

set_lg_bsize 2097152

set_lk_detect DB_LOCK_DEFAULT

set_tmp_dir /tmp

set_lock_timeout 1000

set_txn_timeout 1000

mutex_set_max 65536

Regarding the sizing of virtual machines that host the SKS nodes. I originally provisioned mine as 4 VCPU, 4 GB of RAM, & 50 GB of disk storage (currently showing ~20GB used). However, with the recent additions to my sksconf file to try and address poison keys I found it necessary to increase them to 8 GB of RAM to avoid swapping. I could probably down size the VCPU count, as I rarely see that spike above 50% (out of 400%). I use Ubuntu Bionic (18.04 LTS), only after installing the SKS package that ships with that release I replace it with the version built to refuse at least one of the poison keys as discussed @ https://lists.nongnu.org/archive/html/sks-devel/2018-07/msg00053.html. The actual link to download the package file is @ https://launchpad.net/~canonical-sysadmins/+archive/ubuntu/sks-public/+packages. Hopefully one day it will be promoted into to ship by default without having to download and install manually. 

I think that about covers it, but if you have any questions or notice any mistakes please let me know. Hopefully you and others will find this to be a useful, one-stop-shop resource for setting up a more solid pool of SKS servers.

Cheers,

-T

On Feb 14, 2019, at 4:34 PM, Jonathon Weiss <address@hidden> wrote:

Hello all,

I seem to recall that several key server operators are running in a configuration with multiple SKS instances on a single machine, and others with multiple machines running SKS.

Would anyone doing either of these things be willing to share their configurations (especially sksconf and membership)?

Thanks,
Jonathon

Jonathon Weiss <address@hidden>
MIT/IS&T/Cloud Platforms

_______________________________________________
Sks-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/sks-devel

_______________________________________________
Sks-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/sks-devel