Install PostgreSQL

This guide provides instructions on how to install PostgreSQL for MiaRec.

Preparing The System

Update system default applications:

sudo apt-get update

Configure PostgreSQL Apt repository

These instructions are based on http://www.postgresql.org/download/linux/ubuntu/

Postgres is included into default repository of Ubuntu LTS, but its version is not up to date.

Create the file /etc/apt/sources.list.d/pgdg.list and add a line for the repository:

deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main

Import the repository signing key, and update the package lists:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
sudo apt-key add -
sudo apt-get update

Then install the required version.

sudo apt-get install postgresql-9.4 postgresql-contrib-9.4

Create MiaRec user and database

The default database name and database user are postgres. Switch to postgres user to perform the following operations.

In this example we create user 'miarec' and database 'miarecdb'.

First, start psql command-line interface as postgres user:

sudo -u postgres psql postgres

Create user for MiaRec application:

CREATE USER miarec PASSWORD 'password';

Create MiaRec database:

CREATE DATABASE miarecdb WITH ENCODING 'UNICODE' LC_COLLATE 'C' LC_CTYPE 'C' TEMPLATE template0;
ALTER DATABASE miarecdb OWNER TO miarec;

Connect to "miarecdb" database:

\c miarecdb;

Install uuid-ossp and hstore extensions into "miarecdb" database:

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "hstore";

Enter \q to exit from psql command-line interface:

\q

PostgreSQL Configuration

The postgresql server is using two main configuration files

• /etc/postgresql/9.4/main/pg_hba.conf
• /etc/postgresql/9.4/main/postgresql.conf

pg_hba.conf

Change authentication method from ident to md5 for localhost connections.

Change:

host    all   all     127.0.0.1/32       ident

To:

host    all   all     127.0.0.1/32        md5

When other MiaRec components are deployed on dedicated servers, then you need to add their ip-addresses to trust group. For example:

host    all   all     192.168.0.10/32    md5      # allow access from 192.168.0.10
host    all   all     192.168.1.1/24     md5      # allow access from network 192.168.1.1/24

postgresql.conf

If other MiaRec components (like recorder and web portal) are deployed on dedicated servers, then you need to configure postgres to accept network connections. Change:

listen_addresses = 'localhost'

to

listen_addresses = '*'

Restart PostgreSQL

service postgresql restart

Install Python

Install python 3 from default repository.

sudo apt-get install python3

Execute the following command and make sure that it is at lease version 3.4:

python3 --version

If it is older, then you need to install python 3 manually from sources.

Install Apache Web Server

Install Apache web server and required packages

sudo apt-get install apache2 apache2-dev openssl libssl-dev

Start Apache web server

sudo service apache2 start

Install Redis cache

Download Redis from http://redis.io/download:

wget http://download.redis.io/releases/redis-3.2.1.tar.gz

Extract it and compile with:

tar -xzvf redis-3.2.1.tar.gz
cd redis-3.2.1
make

Install binaries:

sudo make install

Create init script for redis

• Create a directory where to store your Redis config files and your data:

  sudo mkdir /etc/redis
  sudo mkdir /var/redis
  

• Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:

  sudo cp utils/redis_init_script /etc/init.d/redis_6379
  

• Edit the init script.

  sudo vim /etc/init.d/redis_6379
  

  Add the following lines at the top of init script (below line #!/bin/sh):

  #!/bin/sh
  #
  # Simple Redis init.d script conceived to work on Linux systems
  # as it does use of the /proc filesystem.
  #
  # chkconfig:   - 85 15
  # description:  Redis is a persistent key-value database
  #
  # processname: redis
  ### BEGIN INIT INFO
  # Provides: redis_6379
  # Required-Start:    $network $remote_fs $local_fs
  # Required-Stop:     $network $remote_fs $local_fs
  # Default-Start:     2 3 4 5
  # Default-Stop:      0 1 6
  # Short-Description: start and stop redis_6379
  # Description: Redis daemon
  ### END INIT INFO
  

  Make sure to modify REDIS_PORT accordingly to the port you are using. Both the pid file path and the configuration file name depend on the port number.

• Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:

  sudo cp redis.conf /etc/redis/6379.conf
  

• Create a directory inside /var/redis that will work as data and working directory for this Redis instance:

  sudo mkdir /var/redis/6379
  

• Edit the configuration file, making sure to perform the following changes:

  sudo vim /etc/redis/6379.conf
  

  • Set daemonize to yes (by default it is set to no).

    # By default Redis does not run as a daemon. Use 'yes' if you need it.
    # Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
    
    daemonize yes
    

  • Set the pidfile to /var/run/redis_6379.pid (modify the port if needed).

    # If a pid file is specified, Redis writes it where specified at startup
    # and removes it at exit.
    #
    # When the server runs non daemonized, no pid file is created if none is
    # specified in the configuration. When the server is daemonized, the pid file
    # is used even if not specified, defaulting to "/var/run/redis.pid".
    #
    # Creating a pid file is best effort: if Redis is not able to create it
    # nothing bad happens, the server will start and run normally.
    
    pidfile /var/run/redis_6379.pid
    

  • Change the port accordingly. In our example it is not needed as the default port is already 6379.

    # Accept connections on the specified port, default is 6379 (IANA #815344).
    # If port 0 is specified Redis will not listen on a TCP socket.
    
    port 6379
    

  • Set your preferred loglevel.

    # Specify the server verbosity level.
    # This can be one of:
    # debug (a lot of information, useful for development/testing)
    # verbose (many rarely useful info, but not a mess like the debug level)
    # notice (moderately verbose, what you want in production probably)
    # warning (only very important / critical messages are logged)
    
    loglevel notice
    

  • Set the logfile to /var/log/redis_6379.log

    # Specify the log file name. Also the empty string can be used to force
    # Redis to log on the standard output. Note that if you use standard
    # output for logging but daemonize, logs will be sent to /dev/null
    
    logfile "/var/log/redis_6379.log"
    

  • Set the dir to /var/redis/6379

    # The working directory.
    #
    # The DB will be written inside this directory, with the filename specified
    # above using the 'dbfilename' configuration directive.
    #
    # The Append Only File will also be created inside this directory.
    #
    # Note that you must specify a directory here, not a file name.
    
    dir /var/redis/6379
    

  • Uncomment line # bind 127.0.0.1 (very important step for security reasons! With such settings redis will be accessible only from localhost. It will reject connections from outside network.)

    # ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
    # internet, binding to all the interfaces is dangerous and will expose the
    # instance to everybody on the internet. So by default we uncomment the
    # following bind directive, that will force Redis to listen only into
    # the IPv4 lookback interface address (this means Redis will be able to
    # accept connections only from clients running into the same computer it
    # is running).
    #
    # IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
    # JUST COMMENT THE FOLLOWING LINE.
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    bind 127.0.0.1
    

• Finally add the new Redis init script to all the default runlevels using the following command:

  sudo update-rc.d redis_6379 defaults
  

You are done! Now you can try running your instance with:

sudo /etc/init.d/redis_6379 start

Make sure that everything is working as expected:

• Try pinging your instance with redis-cli ping.
• Do a test save with redis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
• Check that your Redis instance is correctly logging in the log file /var/log/redis_6379.log.
• If it's a new machine where you can try it without problems make sure that after a reboot everything is still working.

Install MiaRec Web Application

1.1. Installed required packages

sudo apt-get install libpq-dev gcc libffi-dev libssl-dev python3-setuptools python3-dev

1.2. Create A Python Virtual Environment

It is best practice to install MiaRec web application into a "virtual" Python environment in order to obtain isolation from any "system" packages you've got installed in your Python version. This can be done by using the virtualenv package. Using a virtualenv will also prevent MiaRec from globally installing versions of packages that are not compatible with your system Python.

Install python pip package:

sudo apt-get install python3-pip

Install python venv package (modify python version in this command according to your python version, see output of python3 --version):

sudo apt-get install python3.4-venv

Create virtual environment:

sudo apt-get install python3-venv

sudo easy_install3 virtualenv

Create virtual environment:

mkdir /var/www/miarec
python3 -m venv /var/www/miarec/pyenv

1.3. Install MiaRec Web Application

Contact us (sales@miarec.com) to receive URL to MiaRec installation files.

Download MiaRec web application archive:

wget CONTACT_US_FOR_URL

Extract:

tar -xzvf miarecweb*.tar.gz

Move it to /var/www/miarec/app

mv miarecweb-*/ /var/www/miarec/app

Activate virtual environment:

sudo su
source /var/www/miarec/pyenv/bin/activate

Note, sudo su command is necessary because by default /var/www/ directory is not writable for non-root users in Ubuntu. We are going to install MiaRec web portal files into that directory. That's why you need to switch to root account now.

Upgrade pip to the latest version:

pip install --upgrade pip

Install MiaRec web application into python environment:

pip install -e /var/www/miarec/app

Create log and cache directories for MiaRec web application:

mkdir /var/log/miarecweb
mkdir /var/www/miarec/cache

Make Apache an owner of theses directory. So, it can create log and cache files there.

chown www-data:www-data /var/log/miarecweb
chown www-data:www-data /var/www/miarec/cache

2. Configure MiaRec Web Portal Application

Copy production.ini file from a sample file:

cp /var/www/miarec/app/production.ini.sample /var/www/miarec/production.ini

Edit production.ini file:

vim /var/www/miarec/production.ini

Change in this file the following parameters according to previously installed PostgreSQL and Redis:

• DATABASE_HOST (use 127.0.0.1 if the database is installed on the same host)
• DATABASE_PORT (default is 5432)
• DATABASE_NAME (should match to the previously created database name, default is miarecdb)
• DATABASE_USER (should match to previously created database user for miarec, default is miarec)
• DATABASE_PASSWORD (should match to previously created database user password)
• REDIS_HOST (use 127.0.0.1 if Redis is installed on the same host)
• REDIS_PORT (default is 6379)

If the Redis service is configured with a non-default port (which is 6379), then replace 6379 with the appropriate port number. If the Redis service is running on a dedicated server, then replace 127.0.0.1 to appropriate ip-address.

3. Initialize MiaRec database layout

source /var/www/miarec/pyenv/bin/activate

alembic -c /var/www/miarec/production.ini upgrade head

4. Install Apache mod_wsgi module.

sudo apt-get install libapache2-mod-wsgi-py3

Copy miarec.wsgi.sample into miarec.wsgi

cp /var/www/miarec/app/miarec.wsgi.sample /var/www/miarec/miarec.wsgi

5. Edit Apache configuration file

Create miarec.conf file inside /etc/apache2/sites-available directory:

vi /etc/apache2/sites-available/miarec.conf

Content of this file:

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=www-data group=www-data python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec
WSGIProcessGroup miarec

<Directory /var/www/miarec/app>
  Require all granted
</Directory>

Disable a default site:

a2dissite 000-default

Enable miarec site:

a2ensite miarec

Restart Apache:

service apache2 reload

6. Access MiaRec Web Portal With Web Browser

Now you should be accessed MiaRec from the web browser with the URL http://

You may need to configure firewall exception rules on the server to allow inbound connections to the server on port 80.

Install Celery task manager

Celery is an asynchronous task queue/job queue system, which is used by the MiaRec web portal for different tasks (long-running tasks and/or periodic jobs).

Celery itself is already installed on your system when you deployed MiaRecWeb portal. The only missing part is to run Celery as a daemon.

There are two celery daemons:

• Celery worker executes long-running jobs like call backup/restore, purge deleted records etc.
• Celery scheduler manages periodic tasks. It loads job schedule configuration from MiaRec and initiates execution of these jobs by Celery worker at regular intervals.

The current document is based on official celery documentation.

1. Celery worker daemon (celeryd)

1.1. Create init.d startup script

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celeryd

Make it executable:

chmod +x celeryd

Move to /etc/init.d/

mv celeryd /etc/init.d/

1.2. Create celery worker configuration file

vi /etc/default/celeryd

Content of this file should be:

# Names of nodes to start
CELERYD_NODES="worker1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYD_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8 --ini-file=/var/www/miarec/production.ini"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/miarec/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYD_USER="root"
CELERYD_GROUP="root"

1.3. Install this init.d script and configure it to start automatically during boot process

update-rc.d celeryd defaults

1.4. Start celery

service celeryd start

2. Celery scheduler daemon (celerybeat)

2.1. Create init.d startup script for celery scheduler

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celerybeat

Make it executable:

chmod +x celerybeat

Move to /etc/init.d/

mv celerybeat /etc/init.d/

2.2. Create celery scheduler configuration file

vi /etc/default/celerybeat

Content of this file should be:

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYBEAT_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the scheduler
CELERYBEAT_OPTS="-S miarecweb.jobs.scheduler.JobScheduler --ini-file /var/www/miarec/production.ini"

CELERYBEAT_LOG_FILE="/var/log/miarec/celery/beat.log"
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYBEAT_USER="root"
CELERYBEAT_GROUP="root"

2.3. Install this init.d script and configure it to start automatically during boot process

update-rc.d celerybeat defaults

2.4. Start celery beat

service celerybeat start