Install Celery task manager

Celery is an asynchronous task queue/job queue system, which is used by MiaRec web portal for different tasks (long running tasks and/or periodic jobs).

Celery itself is already installed on your system when you deployed MiaRecWeb portal. The only missing part is to run Celery as a daemon.

There are two celery daemons:

  • Celery worker executes long-running jobs like call backup/restore, purge deleted records etc.
  • Celery scheduler manages periodic tasks. It loads job schedule configuration from MiaRec and initiates execution of these jobs by Celery worker at regular intervals.

The current document is based on official celery documentation.

1. Celery worker daemon (celeryd)

1.1. Create init.d startup script

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celeryd

Make it executable:

chmod +x celeryd

Move to /etc/init.d/

mv celeryd /etc/init.d/

1.2. Create celery worker configuration file

vi /etc/default/celeryd

Content of this file should be:

# Names of nodes to start
CELERYD_NODES="worker1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYD_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 -Ofair --concurrency=8 --ini-file=/var/www/miarec/production.ini"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/miarec/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYD_LOG_LEVEL="WARN"

CELERYD_USER="root"
CELERYD_GROUP="root"

1.3. Install this init.d script and configure it to start automatically during boot process

chkconfig --add celeryd
chkconfig celeryd on

1.5. Start celery

service celeryd start

2. Celery scheduler daemon (celerybeat)

2.1. Create init.d startup script for celery scheduler

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celerybeat

Make it executable:

chmod +x celerybeat

Move to /etc/init.d/

mv celerybeat /etc/init.d/

2.2. Create celery scheduler configuration file

vi /etc/default/celerybeat

Content of this file should be:

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYBEAT_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the scheduler
CELERYBEAT_OPTS="-S miarecweb.jobs.scheduler.JobScheduler --ini-file /var/www/miarec/production.ini"

CELERYBEAT_LOG_FILE="/var/log/miarec/celery/beat.log"
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYBEAT_LOG_LEVEL="WARN"

CELERYBEAT_USER="root"
CELERYBEAT_GROUP="root"

2.3. Install this init.d script and configure it to start automatically during boot process

chkconfig --add celerybeat
chkconfig celerybeat on

2.4. Start celery beat

service celerybeat start

2.5. Configure logrotate to automatically delete old logs

Create file /etc/logrotate.d/celery with the following content:

/var/log/miarec/celery/*.log {
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
    daily
    dateext
    rotate 7
    size 10M
}