Enterprise On-Premise

Install & manage your on-premise instance of Kantree

Scaling your Kantree deployment

Understanding the different components of Kantree

Kantree is divided into 4 components:

  • web: the web application server (port 5000 by default)
  • worker: background worker
  • scheduler: the process launching tasks at fixed time
  • push: the push server, a socket.io server used to send events from the server to the client as they happen (port 4000 by default)

IMPORTANT: Only 1 scheduler instance can be started at a time across all your machines

Application processes

Kantree uses Gunicorn to run its application processes. The command line to launch gunicorn is found in the Procfile.

You can adjust the number of workers (4 by default) based on your machine processing power (CPU). A common formula for the number of workers you shoud use is 2x{number of CPU cores}+1.

Background processing

Kantree makes extensive use of background processing. Queues can accumulate tasks at time and properly scaling workers is important.

Increasing the number of workers

You can increase the number of workers by increasing the number of “worker” processes when using the process manager (via the Procfile). Example using our built-in process manager:

$ ./platform run -c worker=NUMBER_OF_WORKERS

The formula 2x{number of CPU cores}+1 can also be used to calculate the number of possible workers. But the sum total of application and background workers can not exceed this formula.

Using multiple queues

Tasks can be dispatched in up to four different queues. By default, a single queue is used for all tasks. To use different queues for some tasks, use the following config key in your configuration file:

queue_project_hooks: QUEUE_NAME
queue_compute_formulas: QUEUE_NAME
queue_refresh_charts: QUEUE_NAME
queue_batch_actions: QUEUE_NAME

It is good practice to at least use a dedicated queue for queue_compute_formulas and queue_refresh_charts once you are seeing a moderate usage of Kantree.

queue_compute_formulas: low
queue_refresh_charts: low

Once you have configured the queue names in the configuration file, you need to ensure that some workers will process them. By default, workers only process tasks from the default queue. To launch a worker for a specific queue, you will need to tweak the Procfile. Example adding a worker to process the low queue we have mentionned before:

web: gunicorn --name kantree --workers 4 --log-file=- --access-logfile=- --bind 0.0.0.0:5000 wsgi:app
push: flask push-server
worker: flask rq worker
worker_low: flask rq worker low
scheduler: flask rq scheduler

Once configured, if you want to increase the number of workers for a queue:

$ ./platform run -c worker_low=NUMBER_OF_WORKERS

Real-time connections

Real-time connections require one socket per connection. On linux, this means one file descriptor per connection. The default max opened file is 1024 which can quickly become a bottleneck.

Increase the limit using the fs.file-max kernel directive:

# sysctl -w fs.file-max=MAX_FILE

Scaling real-time servers can be hard. If you are serving thousands of users, contact us for support.

Multi-node deployment

Kantree’s architecture makes it very easy to deploy on multiple machines.

You can deploy as many instances of Kantree processes as you want on as many machine as long as they connect to the same PostgreSQL and Redis instances.

Installing a node

First, generate a configuration file that you can use on all your nodes:

$ ./platform gen_config config-prod.yml

Then you can unpack the kantree archive on each node, add your configuration file and your license file and then run:

$ ./platform init_node

Upgrade your database from one of the node using:

$ ./platform upgrade_db

To start only a specific process, use the run command with the process name as argument:

$ ./platform run web
$ ./platform run worker
$ ./platform run scheduler
$ ./platform run push

You don’t have to start all the processes together. You can for example start worker processes on one machine and web and push processes on another.

If you are starting workers on a machine dedicated to background processing, remember to properly scale the number of workers. Example:

$ ./platform run -c worker=NUMBER_OF_WORKERS worker

Load balacing the nodes

Finally, you’ll need to install a load-balancer to serve Kantree using all the nodes. We recommend Nginx (which we use ourselves). You will find an nginx.conf.example file at the root of your installation.

Add the ip addresses for all your nodes running web processes in the upstream app section. Same of the push processes under the upstream push section.

You will also need to replace the {{SERVER_NAME}} and {{PATH_TO_KANTREE_ROOT}} placeholders.

IMPORTANT: for the push server, the session needs to be sticky (ie. the requests coming from a client needs to always reach the same server).

Services Kantree depends on

Redis

Given the usage we do of Redis, you shouldn’t need to have a complicated Redis deployement. For almost all setups, Redis can be installed on the same machine as your Kantree server. If the application server becomes to low on RAM, you can put Redis on its own machine and that should be enough.

In case you want to ensure high-avaibility, you can check out Redis Sentinel.

Postgresql

Postgresql can be a tricky beast to scale. For the vast majority of deployments, Postgresql can live on a single machine. If you encounter performance issues due to the database, consider upgrading the machine itself (the hardware) before trying to scale horizontally.

There are many ressources to help you scale Postgresql or configure high-availability. It is not the purpose of this guide to cover these aspects.