[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Sks-devel] SKS scaling configuration
From: |
Jeremy T. Bouse |
Subject: |
Re: [Sks-devel] SKS scaling configuration |
Date: |
Mon, 4 Mar 2019 18:48:29 -0500 |
User-agent: |
Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 |
I'd previously had only 2 instances and if they weren't peering outside
and one went down it seemed to cause problems. I went with 3 backend
secondary nodes with the primary node doing the peering outside my
network this time since I was re-deploying from scratch. This way I can
take 1 node out and the cluster still has 3 active nodes. I do have my
primary shutting down to perform key dumps and I also have more of the
traffic for queries go to the secondary nodes and leave the primary node
to handle the peering recon. In fact I just dumped the keys from my
primary node and rebuilt all 4 nodes from the dump last night.
On 3/4/2019 6:18 PM, Jonathon Weiss wrote:
> Thanks for the information everyone. A further question: I saw the
> advice of a minimum of three servers. Anyone know how that was
> arrived at, or if there is a recommendation on how many queries an
> individual SKS back-end can handle?
>
> Jonathon
>
> Jonathon Weiss <address@hidden>
> MIT/IS&T/Cloud Platforms
>
>
>
>
> On Fri, 1 Mar 2019, Jeremy T. Bouse wrote:
>
>>
>> I ended up with the following NGINX configuration...
>>
>> in /etc/nginx/conf.d/upstream.conf:
>>
>> upstream sks_secondary {
>> server 127.0.0.1:11371 weight=5;
>> server 172.16.20.52:11371 weight=10;
>> server 172.16.20.53:11371 weight=10;
>> server 172.16.20.54:11371 weight=10;
>> }
>>
>> upstream sks_primary {
>> server 127.0.0.1:11371;
>> server 172.16.20.52:11371 backup;
>> server 172.16.20.53:11371 backup;
>> server 172.16.20.54:11371 backup;
>> }
>>
>> map $arg_op $sks_server {
>> "stats" sks_primary;
>> default sks_secondary;
>> }
>>
>> in /etc/nginx/site-available/sks-default:
>>
>> server {
>> listen 172.16.20.51:80 default_server;
>> listen 172.16.20.51:11371 default_server;
>> listen [::]:80 ipv6only=on default_server;
>> # listen [::]:11371 ipv6only=on default_server;
>> access_log off;
>> server_tokens off;
>> root /var/www/html;
>> index index.html index.htm;
>>
>> proxy_buffering on;
>> proxy_buffer_size 1k;
>> proxy_buffers 24 4k;
>> proxy_busy_buffers_size 8k;
>> proxy_max_temp_file_size 2048m;
>> proxy_temp_file_write_size 32k;
>>
>> location /pks/hashquery {
>> proxy_ignore_client_abort on;
>> proxy_pass http://$sks_server;
>> proxy_set_header Host $host;
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_set_header X-Forwarded-Port $server_port;
>> proxy_pass_header Server;
>> add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>> }
>>
>> location /pks/add {
>> proxy_ignore_client_abort on;
>> proxy_pass http://$sks_server;
>> proxy_set_header Host $host;
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_set_header X-Forwarded-Port $server_port;
>> proxy_pass_header Server;
>> add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>> client_max_body_size 8m;
>> }
>>
>> location /pks {
>> proxy_cache ugns_sks_cache;
>> # proxy_cache_background_update on;
>> proxy_cache_lock on;
>> proxy_cache_min_uses 3;
>> proxy_cache_revalidate on;
>> proxy_cache_use_stale error timeout updating http_500
>> http_502 http_503 http_504;
>> proxy_cache_valid any 10m;
>> proxy_ignore_client_abort on;
>> proxy_ignore_headers Cache-Control "Expires";
>> proxy_pass http://$sks_server;
>> proxy_set_header Host $host;
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_set_header X-Forwarded-Port $server_port;
>> proxy_pass_header Server;
>> add_header Via "1.1 sks.undergrid.net:$server_port (nginx)";
>> add_header X-Proxy-Cache $upstream_cache_status;
>> }
>> }
>>
>> The NGINX configuration appears to be working fine for me... My 3
>> backend nodes are operating as I expect as well.. The problem I'm
>> seeing exhibited currently is that my primary node which is running
>> along with NGINX seems to be
>> writing an awful lot of log files and not processing them despite
>> having the same DB_CONFIG file as the other 3 nodes. With each of the
>> log files running 100MB it is quickly filling up the drive on that
>> node and then the sks and
>> sks-recon processes simply fall over and crash but the other 3 nodes
>> running behind it keep chugging along just not getting any further
>> gossip input.
>>
>> I keep seeing the following log entry popping up only on my primary
>> node:
>>
>> add_keys_merge failed: Eventloop.SigAlarm
>>
>> On 2/25/2019 12:37 PM, Todd Fleisher wrote:
>> On Feb 23, 2019, at 8:35 PM, Jeremy T. Bouse
>> <address@hidden> wrote:
>>
>> I didn't have as many locations configured as you show in your
>> example but it looked like you were defining the map but I didn't see
>> it being used in any of your location blocks unless I'm missing
>> something. Shouldn't you
>> be using $upstream_server in your proxy_pass configuration?
>>
>> I think you’re on to something here. I just tried commenting out the
>> other servers from the upstream sks_servers_primary block and am
>> still seeing stats queries hitting the commented out servers.
>> Kristian - could you please double check the configuration snippets
>> you provided to me last year and see if something was missing related
>> to this?
>>
>> -T