Scaling your Kantree deployment

Understanding the different components of Kantree

Kantree is divided into the following components:

• web: the web application server (port 5000)
• push: the realtime server, a socket.io server pushing updates to the browsers (port 4000)
• collaboration: the collaboration server (port 4500)
• mcp: the MCP server, when mcp_enabled is set (port 4600)
• worker: background worker
• scheduler: the process launching tasks at fixed time

By default, the app container starts all of them.

IMPORTANT: Only 1 scheduler instance can be started at a time across all your machines

Running specific processes

When started without any command, the container launches all the processes. Use the run command to start only some of them, and to adjust their number. With Compose, set it as the command of a service which uses the same image, configuration and volumes as the app service:

services:
  worker:
    # same image, configuration and volumes as the app service
    command: run worker --worker-scale 4

The processes are named web, push, worker and scheduler.

Note that starting processes this way does not upgrade the database. Run this command after each update:

$ docker compose run --rm app upgrade-db

When the collaboration server is not started in the same container as the other processes, set collab_secret and collab_server_internal_url (see Collaboration server). The collaboration server reaches Kantree at collab_backend_url (default: http://localhost:5000). collab_server_timeout (default: 30) is the timeout, in seconds, of the requests to the collaboration server.

Application processes

Kantree uses Gunicorn to run its application processes.

You can adjust the number of Gunicorn workers based on your machine processing power (CPU). The default number is 2x{number of CPU cores}+1.

Use the --web-scale parameter to modify the number of Gunicorn workers:

run --web-scale NUMBER_OF_WORKERS

Background processing

Kantree makes extensive use of background processing. Queues can accumulate tasks at time and properly scaling workers is important.

Increasing the number of workers

Use the --worker-scale parameter:

run --worker-scale NUMBER_OF_WORKERS

The formula 2x{number of CPU cores}+1 can also be used to calculate the number of possible workers. But the sum total of application and background workers can not exceed this formula.

Using multiple queues

Tasks can be dispatched in different queues. By default, a single queue is used for all tasks. To use different queues for some tasks, use the following config key in your configuration file:

queue_project_hooks: QUEUE_NAME
queue_automations: QUEUE_NAME
queue_related_cards: QUEUE_NAME
queue_compute_formulas: QUEUE_NAME
queue_refresh_charts: QUEUE_NAME
queue_batch_actions: QUEUE_NAME
queue_notifications: QUEUE_NAME

It is good practice to at least use a dedicated queue for queue_compute_formulas, queue_refresh_charts and queue_automations once you are seeing a moderate usage of Kantree.

queue_automations: automations
queue_related_cards: low
queue_compute_formulas: low
queue_refresh_charts: low

Once you have configured the queue names in the configuration file, you need to ensure that some workers will process them. By default, workers only process tasks from the default queue. Use the --worker-queues parameter with a comma separated list of queue names:

run --worker-queues default,low,automations

This will create one worker per queue. If you combine this parameter with --worker-scale X it will start X workers per queue.

You can use the --worker-combine-queues paremeter to instead launch X workers that each process tasks from all queues.

Timeouts

Background tasks are stopped after tasks_timeout seconds (default: 180), and scheduled tasks after tasks_scheduled_timeout seconds (default: 300).

Real-time connections

Real-time connections require one socket per connection. On linux, this means one file descriptor per connection. The default max opened file is 1024 which can quickly become a bottleneck.

Increase the limit using the fs.file-max kernel directive on the host:

# sysctl -w fs.file-max=MAX_FILE

and the nofile limit of the container (ulimits in Compose).

Scaling real-time servers can be hard. If you are serving thousands of users, contact us for support.

Multi-node deployment

Kantree’s architecture makes it very easy to deploy on multiple machines.

You can run as many Kantree containers as you want on as many machine as long as they connect to the same PostgreSQL and Redis instances, and share the same configuration.

Uploaded files must be available to all the containers: use a shared volume or Amazon S3.

Remember that only one scheduler must run: start the other containers with the processes they need (see Running specific processes). For example, you can start worker processes on one machine and web and push processes on another.

If you are starting workers on a machine dedicated to background processing, remember to properly scale the number of workers.

Load balacing the nodes

Finally, you’ll need a load-balancer to serve Kantree using all the nodes. We recommend Nginx (which we use ourselves): start from the nginx-docker.conf file of the example.

List the addresses of all your nodes running web processes in the upstream of the web server, and the ones running push processes in the upstream of the push server. The paths and ports to forward are listed in Using your own reverse proxy.

IMPORTANT: for the realtime server, the session needs to be sticky (ie. the requests coming from a client needs to always reach the same server).

Services Kantree depends on

Redis

Given the usage we do of Redis, you shouldn’t need to have a complicated Redis deployement. For almost all setups, Redis can be installed on the same machine as your Kantree server. If the application server becomes to low on RAM, you can put Redis on its own machine and that should be enough.

In case you want to ensure high-avaibility, you can check out Redis Sentinel.

If several Kantree instances (not the nodes of one instance) use the same Redis server, give each instance a different push_channel (default: socketio).

Postgresql

Postgresql can be a tricky beast to scale. For the vast majority of deployments, Postgresql can live on a single machine. If you encounter performance issues due to the database, consider upgrading the machine itself (the hardware) before trying to scale horizontally.

There are many ressources to help you scale Postgresql or configure high-availability. It is not the purpose of this guide to cover these aspects.