Install PostgreSQL

This guide provides instructions how to install PostgreSQL for MiaRec.

Preparing The System

Update system default applications:

yum update

Configure your YUM repository

Postgres is included into default repository of CentOS / Red Hat / Fedora, but its version is not up to date.

Locate and edit your distributions .repo file, located:

• On Fedora: /etc/yum.repos.d/fedora.repo and /etc/yum.repos.d/fedora-updates.repo, [fedora] sections
• On CentOS: /etc/yum.repos.d/CentOS-Base.repo, [base] and [updates] sections
• On Red Hat: /etc/yum/pluginconf.d/rhnplugin.conf [main] section

To the section(s) identified above, you need to append a line (otherwise dependencies might resolve to the postgresql supplied by the base repository):

exclude=postgresql*

For example, on Centos 6 the .repo file should be:

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
exclude=postgresql*

#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
exclude=postgresql*

Install PGDG RPM file

A PGDG file is available for each distribution/architecture/database version combination. Browse http://yum.postgresql.org and find your correct RPM. For example, to install PostgreSQL 9.4 on CentOS 6 64-bit:

Centos 6:

yum localinstall http://yum.postgresql.org/9.4/redhat/rhel-6-x86_64/pgdg-centos94-9.4-2.noarch.rpm

Centos 7:

yum localinstall http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-2.noarch.rpm

Install PostgreSQL database server and its extensions

yum install postgresql94-server postgresql94-contrib

Initialize data directory

After installing the packages, a database needs to be initialized and configured.

service postgresql-9.4 initdb

If the previous command did not work, try directly calling the setup binary, located in:

/usr/pgsql-9.4/bin/postgresql94-setup initdb

Configure start at boot up

chkconfig postgresql-9.4 on

Start PostgreSQL database service

service postgresql-9.4 start

Create MiaRec user and database

In this example we create user 'miarec' and database 'miarecdb'.

First, connect/login as root:

sudo -u postgres psql postgres

This will open psql command-line interface.

Create user (replace 'password' with something more secure):

CREATE USER miarec PASSWORD 'password';

Create database:

CREATE DATABASE miarecdb WITH ENCODING 'UNICODE' LC_COLLATE 'C' LC_CTYPE 'C' TEMPLATE template0;
ALTER DATABASE miarecdb OWNER TO miarec;

Connect to "miarecdb" database:

\c miarecdb;

Install uuid-ossp and hstore extensions into "miarecdb" database:

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "hstore";

Enter \q to exit from psql command-line interface:

\q

PostgreSQL Configuration

The postgresql server is using two main configuration files

• /var/lib/pgsql/9.4/data/pg_hba.conf
• /var/lib/pgsql/9.4/data/postgresql.conf

Edit pg_hba.conf

vi /var/lib/pgsql/9.4/data/pg_hba.conf

Change authentication method from ident to md5 for localhost connections.

Change:

host    all   all     127.0.0.1/32       ident

To:

host    all   all     127.0.0.1/32        md5

When other MiaRec components are deployed on dedicated servers, then you need to add their ip-addresses to trust group. For example:

host    all   all     192.168.0.10/32    md5      # allow access from 192.168.0.10
host    all   all     192.168.1.1/24     md5      # allow access from network 192.168.1.1/24

Edit postgresql.conf (optional)

vi /var/lib/pgsql/9.4/data/postgresql.conf

If other MiaRec components (like recorder and web portal) are deployed on dedicated servers, then you need to configure postgres to accept inbound network connections. Change:

listen_addresses = 'localhost'

to

listen_addresses = '*'

Restart PostgreSQL

service postgresql-9.4 restart

Install Python 3

1. Preparing The System

Update system default applications:

yum update

Install required packages for building python:

yum install openssl-devel sqlite-devel bzip2-devel gcc wget

2. Build python 3 from source code

Download latest stable Python 3 source code files from https://www.python.org/downloads/source/. MiaRecWeb was tested with Python v.3.4

wget https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz

Extract source code:

tar -xzvf Python-3.*.tgz

Build Python binaries:

cd Python-3*
./configure --enable-shared
make

Option --enable-shared is necessary in order to create shared library *.so files, which will be used later when compiling Apache mod_wsgi module.

3. Install Python

Normally, one would use “make install”; however, in order not to override system defaults - replacing the Python already used by the system - we will use make altinstall.

make altinstall

This will install python into /usr/local/bin with name, which contains version, like /usr/local/bin/python3.4.

4. Add Python shared libraries to PATH

By default Python installed *.so files into /usr/local/lib On some RedHat-based systems this directory is not searched when loading shared libraries. To fix that, you need to edit file /etc/ld.so.conf.

vi /etc/ld.so.conf

Add the following line to that file:

/usr/local/lib

Run ldconfig:

ldconfig

Install Apache web server

Install Apache web server and required packages

yum install httpd httpd-devel openssl openssl-devel

Configure start at boot up

chkconfig httpd on

Start Apache web server

service httpd start

Install Redis cache

Download Redis from http://redis.io/download:

wget http://download.redis.io/releases/redis-3.0.0.tar.gz

Extract it and compile with:

tar -xzvf redis-3.0.0.tar.gz
cd redis-3.0.0
make

Install binaries:

make install

Create data directory for redis

This directory will work as data and working directory for your Redis instance:

mkdir -p /var/redis/6379

Create configuration file for redis

Create a directory where to store your Redis config files:

mkdir /etc/redis

Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:

cp redis.conf /etc/redis/6379.conf

Edit this file, making sure to perform the following changes:

vi /etc/redis/6379.conf

• Set daemonize to yes (by default it is set to no).
• Set the pidfile to /var/run/redis_6379.pid (modify the port if needed).
• Change the port accordingly. In our example it is not needed as the default port is already 6379.
• Set your preferred loglevel.
• Set the logfile to /var/log/redis_6379.log
• Set the dir to /var/redis/6379 (very important step!)
• Uncomment line # bind 127.0.0.1 (very important step for security reasons! With such settings redis will be accessible only from localhost. It will reject connections from outside network.)

Create init script for redis

• Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:

  cp utils/redis_init_script /etc/init.d/redis_6379
  

• Edit the init script.

  vi /etc/init.d/redis_6379
  

  Add the following lines at the top of init script (below line #!/bin/sh):

  
  #!/bin/sh
  #
  # Simple Redis init.d script conceived to work on Linux systems
  # as it does use of the /proc filesystem.
  #
  # chkconfig:   - 85 15
  # description:  Redis is a persistent key-value database
  # processname: redis
  

  Make sure to modify REDIS_PORT accordingly to the port you are using. Both the pid file path and the configuration file name depend on the port number.

• Finally add the new Redis init script to all the default runlevels using the following command:

  chkconfig --add redis_6379
  chkconfig redis_6379 on
  

You are done! Now you can try running your instance with:

/etc/init.d/redis_6379 start

Verify Redis installation

Make sure that everything is working as expected:

• Try pinging your instance with redis-cli ping.
• Do a test save with redis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
• Check that your Redis instance is correctly logging in the log file /var/log/redis_6379.log.
• If it's a new machine where you can try it without problems make sure that after a reboot everything is still working.

Install MiaRec web application

1.1. Installed required packages

yum install postgresql94-devel gcc libffi-devel openssl-devel libxml2-devel libxslt-devel

1.2. Create python virtual environment

It is best practice to install MiaRec web application into a "virtual" Python environment in order to obtain isolation from any "system" packages you've got installed in your Python version. This can be done by using the virtualenv package. Using a virtualenv will also prevent MiaRec from globally installing versions of packages that are not compatible with your system Python.

Upgrade pip and setuptools to the latest version:

python3 -m pip install -U pip setuptools

On Centos 7 you may need to run python3.4 ... instead of python3 ...

Create virtual environment:

mkdir /var/www/miarec
python3 -m venv /var/www/miarec/pyenv

1.3. Install MiaRec web application

Fill the download form to request URL to MiaRec Web portal installation files.

Download MiaRec web application archive:

wget CONTACT_US_FOR_URL

Extract:

tar -xzvf miarecweb*.tar.gz

Move it to /var/www/miarec/app

mv miarecweb-*/ /var/www/miarec/app

Make sure that postgresql pg_config file is in PATH (it is required by psycopg2 python package). Execute in shell:

PATH=$PATH:/usr/pgsql-9.4/bin/

Activate virtual environment:

source /var/www/miarec/pyenv/bin/activate

Upgrade pip to the latest version:

pip install --upgrade pip

Install MiaRec web application into python environment:

pip install -e /var/www/miarec/app

Create log and cache directories for MiaRec web application:

mkdir /var/log/miarecweb
mkdir /var/www/miarec/cache

Make Apache an owner of these directories. So, it can create log and cache files there.

chown apache:apache /var/log/miarecweb
chown apache:apache /var/www/miarec/cache

2. Configure MiaRec web portal application

Create production.ini file from a sample file (production.ini.sample):

cp /var/www/miarec/app/production.ini.sample /var/www/miarec/production.ini

Edit production.ini file:

vi /var/www/miarec/production.ini

Change in this file the following parameters according to previously installed PostgreSQL and Redis:

• DATABASE_HOST (use 127.0.0.1 if database is installed on the same host)
• DATABASE_PORT (default is 5432)
• DATABASE_NAME (should match to previously created database name, default is miarecdb)
• DATABASE_USER (should match to previously created database user for miarec, default is miarec)
• DATABASE_PASSWORD (should match to previously created database user password)
• REDIS_HOST (use 127.0.0.1 if Redis is installed on the same host)
• REDIS_PORT (default is 6379)

If Redis service is configured with non-default port (which is 6379), then replace 6379 with appropriate port number. If Redis service is running on a dedicated server, then replace 127.0.0.1 to appropriate ip-address.

3. Initialize MiaRec database layout

source /var/www/miarec/pyenv/bin/activate

alembic -c /var/www/miarec/production.ini upgrade head

4. Build and install Apache mod_wsgi module.

Download source code of mod_wsgi module from: https://github.com/GrahamDumpleton/mod_wsgi/releases

cd ~
wget https://github.com/GrahamDumpleton/mod_wsgi/archive/4.4.11.tar.gz

Activate previously created python virtual environment. This is necessary because CentOS has older Python version 2.7 and we need to build mod_wsgi for new python 3.4.

source /var/www/miarec/pyenv/bin/activate

Extract, build and install:

tar -xzvf 4.4.11.tar.gz
cd mod_wsgi*
./configure
make
make install

Disable SELinux because it causes Segmentation fault when Apache loads mod_wsgi module.

sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config

Reboot of server is required.

shutdown -r now

5. Edit Apache configuration file

Copy miarec.wsgi.sample into miarec.wsgi. This is a configuration for MiaRec WSGI application, which will be executed inside Apache web server.

cp /var/www/miarec/app/miarec.wsgi.sample /var/www/miarec/miarec.wsgi

Create miarec.conf file inside /etc/httpd/conf.d directory:

vi /etc/httpd/conf.d/miarec.conf

Content of this file:

Apache version 2.2 (for Centos 6):

LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix run/wsgi

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=apache group=apache python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIProcessGroup miarec

Alias /favicon.ico /var/www/miarec/app/miarecweb/static/favicon.ico
Alias /static /var/www/miarec/app/miarecweb/static

WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec

<Directory /var/www/miarec/app/miarecweb/static>
  AllowOverride None
  Order deny,allow
  Allow from all
</Directory>

<Directory /var/www/miarec>
  <Files miarec.wsgi>
      Order deny,allow
      Allow from all
  </Files>
</Directory>

Apache version 2.4 (for Centos 7):

LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix run/wsgi

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=apache group=apache python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIProcessGroup miarec

Alias /favicon.ico /var/www/miarec/app/miarecweb/static/favicon.ico
Alias /static /var/www/miarec/app/miarecweb/static

WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec

<Directory /var/www/miarec/app/miarecweb/static>
  AllowOverride None
  Require all granted
</Directory>

<Directory /var/www/miarec>
  <Files miarec.wsgi>
    Require all granted
  </Files>
</Directory>

Restart Apache:

service httpd restart

6. Access MiaRec web-portal with web-browser

Now you should be access MiaRec from web browser with URL http://ADDRESS

You may need to configure firewall exception rules on the server to allow inbound connections to the server on port 80.

7. Troubleshooting

If you receive "500 Internal Server Error" when accessing MiaRec web portal, then you can check Apache error log file:

less /var/log/httpd/error_log

Install Celery task manager

Celery is an asynchronous task queue/job queue system, which is used by MiaRec web portal for different tasks (long running tasks and/or periodic jobs).

Celery itself is already installed on your system when you deployed MiaRecWeb portal. The only missing part is to run Celery as a daemon.

There are two celery daemons:

• Celery worker executes long-running jobs like call backup/restore, purge deleted records etc.
• Celery scheduler manages periodic tasks. It loads job schedule configuration from MiaRec and initiates execution of these jobs by Celery worker at regular intervals.

The current document is based on official celery documentation.

1. Celery worker daemon (celeryd)

1.1. Create init.d startup script

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celeryd

Make it executable:

chmod +x celeryd

Move to /etc/init.d/

mv celeryd /etc/init.d/

1.2. Create celery worker configuration file

vi /etc/default/celeryd

Content of this file should be:

# Names of nodes to start
CELERYD_NODES="worker1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYD_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 -Ofair --concurrency=8 --ini-file=/var/www/miarec/production.ini"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/miarec/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYD_LOG_LEVEL="WARN"

CELERYD_USER="root"
CELERYD_GROUP="root"

1.3. Install this init.d script and configure it to start automatically during boot process

chkconfig --add celeryd
chkconfig celeryd on

1.5. Start celery

service celeryd start

2. Celery scheduler daemon (celerybeat)

2.1. Create init.d startup script for celery scheduler

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celerybeat

Make it executable:

chmod +x celerybeat

Move to /etc/init.d/

mv celerybeat /etc/init.d/

2.2. Create celery scheduler configuration file

vi /etc/default/celerybeat

Content of this file should be:

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYBEAT_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the scheduler
CELERYBEAT_OPTS="-S miarecweb.jobs.scheduler.JobScheduler --ini-file /var/www/miarec/production.ini"

CELERYBEAT_LOG_FILE="/var/log/miarec/celery/beat.log"
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYBEAT_LOG_LEVEL="WARN"

CELERYBEAT_USER="root"
CELERYBEAT_GROUP="root"

2.3. Install this init.d script and configure it to start automatically during boot process

chkconfig --add celerybeat
chkconfig celerybeat on

2.4. Start celery beat

service celerybeat start

2.5. Configure logrotate to automatically delete old logs

Create file /etc/logrotate.d/celery with the following content:

/var/log/miarec/celery/*.log {
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
    daily
    dateext
    rotate 7
    size 10M
}

yum install libpcap

wget CONTACT_US_FOR_URL

tar -xzvf miarec-*.tar.gz
cd miarec-*

cp miarec /usr/local/bin/

mkdir /etc/miarec
cp miarec.ini /etc/miarec/
cp -r sqlconfig /etc/miarec/

mkdir /var/lib/miarec
cp WebServices.wsdl /var/lib/miarec/

mkdir -p /var/log/miarec

mkdir -p /var/miarec/recordings

vi /etc/miarec/miarec.ini

#-----------------------------------------------------------------
#    SQLConfig
#-----------------------------------------------------------------
#  Loading configuration from SQL database
#-----------------------------------------------------------------
################################################################################
[SQLConfig]
################################################################################

#  Database Driver type.
#  Supported values: 
#    PostgreSQL
#-------------------------------------------------------------------------------
Driver=PostgreSQL
#-------------------------------------------------------------------------------


#  Host of database server
#-------------------------------------------------------------------------------
Host=127.0.0.1:5432
#-------------------------------------------------------------------------------


#  Database name
#-------------------------------------------------------------------------------
Database=miarecdb
#-------------------------------------------------------------------------------


#  Username and password for accessing database. Should have write permissions.
#-------------------------------------------------------------------------------
Username=miarec
Password=password
#-------------------------------------------------------------------------------

#-----------------------------------------------------------------
#  Configuration section for SQLCallsLog module. This module stores calls log into database
#  Supported call events:
#    start,connect,update,stop,stream_start,stream_stop
#-----------------------------------------------------------------
################################################################################
[SQLCallsLog]
################################################################################

#  Database Driver type.
#  Supported values: 
#    PostgreSQL
#-------------------------------------------------------------------------------
Driver=PostgreSQL
#-------------------------------------------------------------------------------


#  Host of database server
#-------------------------------------------------------------------------------
Host=127.0.0.1:5432
#-------------------------------------------------------------------------------


#  Database name
#-------------------------------------------------------------------------------
Database=miarecdb
#-------------------------------------------------------------------------------


#  Username and password for accessing database. Should have write permissions.
#-------------------------------------------------------------------------------
Username=miarec
Password=password
#-------------------------------------------------------------------------------

initctl stop miarec
initctl start miarec

service miarec restart

SystemD start-up script (Centos 7.x)

Create startup script miarec.service in directory /etc/systemd/system/

vi /etc/systemd/system/miarec.service

Content of this file:

[Unit]
Description=MiaRec
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/miarec
ExecStart=/usr/local/bin/miarec --pid /var/run/miarec.pid --core unlimited -c /etc/miarec/miarec.ini
Restart=always
User=root
Group=root
LimitNOFILE=10240
LimitFSIZE=infinity

[Install]
WantedBy=multi-user.target

It is important that WorkingDirectory setting points to directory, where WebServices.wsdl file is located.

Install this startup script:

systemctl enable miarec

Start MiaRec process

systemctl start miarec

Upstart start-up script (Centos 6.x)

Create file /etc/init/miarec.conf

vi /etc/init/miarec.conf

Content of this file:

description "MiaRec call recorder"
author      "MiaRec, Inc. www.miarec.com"

env EXEC=/usr/local/bin/miarec
env PIDFILE=/var/run/miarec.pid
env CONFFILE=/etc/miarec/miarec.ini

start on started sshd
stop on runlevel [!2345]

console output

# Increase open file descriptors limit
limit nofile 10240 10240

# Restart automatically proces in case of crash
respawn

# Stop respawn if it occured more than 10 times during 60 seconds period.
# This means serious problems
respawn limit 10 60

# Current working directory for MiaRec process
chdir /var/lib/miarec

# Enable core dumps for troubleshooting
limit core unlimited unlimited

instance miarec
exec $EXEC -c $CONFFILE --pid $PIDFILE

Reload Upstart configuration

initctl reload-configuration

Validate that miarec is in a list of processes:

initctl list

If you do not see there miarec then check errors in /var/log/messages

Start MiaRec process

initctl start miarec

Stop MiaRec process

initctl stop miarec

Restart MiaRec process

initctl stop miarec
initctl start miarec

Init.d start-up script

Note, this script doesn't support automatic respawn of process in case of abnormal exit (for example, due to app crash). We recommend using alternative methods which support respawn:

• Upstart for RedHat/Centos 6.x
• SystemD for RedHat/Centos 7.x

Create startup script miarec in directory /etc/init.d

vi /etc/init.d/miarec

Content of this file:

#!/bin/sh
#
# Startup sript for MiaRec call recorder
# 
# chkconfig: - 80 20
# description: MiaRec call recorder
# processname: miarec
# config: /etc/miarec.ini
#

EXEC=/usr/local/bin/miarec
PIDFILE=/var/run/miarec.pid
LOCKFILE=/var/lock/subsys/miarec
CONF=/etc/miarec/miarec.ini
CHDIR=/var/lib/miarec
RETVAL=0

# Source function library.
. /etc/rc.d/init.d/functions


start() {
        if [ -f $PIDFILE ]
        then
                RETVAL=1
                echo "$PIDFILE exists, process is already running or crashed" && failure
                echo
                return $RETVAL
        else
                echo "Staring MiaRec recorder..."
                cd $CHDIR
                ulimit -Hn 10240
                ulimit -Sn 10240
                $EXEC -c $CONF --pid $PIDFILE --core unlimited > /dev/null 2>&1 &
                RETVAL=$?
                [ $RETVAL -eq 0 ] && touch $LOCKFILE && success || failure
                echo
                return $RETVAL
        fi
}

stop() {
        echo "Stopping..."
        if [ ! -f $PIDFILE ]
        then
                RETVAL=1
                echo "$PIDFILE does not exist, process is not running" && warning
                echo
            return $RETVAL
        else
                PID=$(cat $PIDFILE)
                kill -TERM $PID
                rm -f $LOCKFILE
                echo "MiaRec stopped" && success
                echo
        fi
}


case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                stop
                start
                ;;
        *)
                echo "Please use start, stop or restart as first argument"
                RETVAL=2
                ;;
esac

exit $RETVAL

Make this script executable:

chmod +x /etc/init.d/miarec

Add it to autostart during boot:

chkconfig --add miarec
chkconfig miarec on

Install PostgreSQL

This guide provides instructions how to install PostgreSQL for MiaRec.

Preparing The System

Update system default applications:

sudo apt-get update

Configure PostgreSQL Apt repository

These instructions are based on http://www.postgresql.org/download/linux/ubuntu/

Postgres is included into default repository of Ubuntu LTS, but its version is not up to date.

Create the file /etc/apt/sources.list.d/pgdg.list and add a line for the repository:

deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main

Import the repository signing key, and update the package lists:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
sudo apt-key add -
sudo apt-get update

Then install the required version.

sudo apt-get install postgresql-9.4 postgresql-contrib-9.4

Create MiaRec user and database

The default database name and database user are postgres. Switch to postgres user to perform the following operations.

In this example we create user 'miarec' and database 'miarecdb'.

First, start psql command-line interface as postgres user:

sudo -u postgres psql postgres

Create user for MiaRec application:

CREATE USER miarec PASSWORD 'password';

Create MiaRec database:

CREATE DATABASE miarecdb WITH ENCODING 'UNICODE' LC_COLLATE 'C' LC_CTYPE 'C' TEMPLATE template0;
ALTER DATABASE miarecdb OWNER TO miarec;

Connect to "miarecdb" database:

\c miarecdb;

Install uuid-ossp and hstore extensions into "miarecdb" database:

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "hstore";

Enter \q to exit from psql command-line interface:

\q

PostgreSQL Configuration

The postgresql server is using two main configuration files

• /etc/postgresql/9.4/main/pg_hba.conf
• /etc/postgresql/9.4/main/postgresql.conf

pg_hba.conf

Change authentication method from ident to md5 for localhost connections.

Change:

host    all   all     127.0.0.1/32       ident

To:

host    all   all     127.0.0.1/32        md5

When other MiaRec components are deployed on dedicated servers, then you need to add their ip-addresses to trust group. For example:

host    all   all     192.168.0.10/32    md5      # allow access from 192.168.0.10
host    all   all     192.168.1.1/24     md5      # allow access from network 192.168.1.1/24

postgresql.conf

If other MiaRec components (like recorder and web portal) are deployed on dedicated servers, then you need to configure postgres to accept network connections. Change:

listen_addresses = 'localhost'

to

listen_addresses = '*'

Restart PostgreSQL

service postgresql restart

Install Python

Install python 3 from default repository.

sudo apt-get install python3

Execute the following command and make sure that it is at lease version 3.4:

python3 --version

If it is older, then you need to install python 3 manually from sources.

Install Apache web server

Install Apache web server and required packages

sudo apt-get install apache2 apache2-dev openssl libssl-dev

Start Apache web server

sudo service apache2 start

Install Redis cache

Download Redis from http://redis.io/download:

wget http://download.redis.io/releases/redis-3.2.1.tar.gz

Extract it and compile with:

tar -xzvf redis-3.2.1.tar.gz
cd redis-3.2.1
make

Install binaries:

sudo make install

Create init script for redis

• Create a directory where to store your Redis config files and your data:

  sudo mkdir /etc/redis
  sudo mkdir /var/redis
  

• Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:

  sudo cp utils/redis_init_script /etc/init.d/redis_6379
  

• Edit the init script.

  sudo vim /etc/init.d/redis_6379
  

  Add the following lines at the top of init script (below line #!/bin/sh):

  #!/bin/sh
  #
  # Simple Redis init.d script conceived to work on Linux systems
  # as it does use of the /proc filesystem.
  #
  # chkconfig:   - 85 15
  # description:  Redis is a persistent key-value database
  #
  # processname: redis
  ### BEGIN INIT INFO
  # Provides: redis_6379
  # Required-Start:    $network $remote_fs $local_fs
  # Required-Stop:     $network $remote_fs $local_fs
  # Default-Start:     2 3 4 5
  # Default-Stop:      0 1 6
  # Short-Description: start and stop redis_6379
  # Description: Redis daemon
  ### END INIT INFO
  

  Make sure to modify REDIS_PORT accordingly to the port you are using. Both the pid file path and the configuration file name depend on the port number.

• Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:

  sudo cp redis.conf /etc/redis/6379.conf
  

• Create a directory inside /var/redis that will work as data and working directory for this Redis instance:

  sudo mkdir /var/redis/6379
  

• Edit the configuration file, making sure to perform the following changes:

  sudo vim /etc/redis/6379.conf
  

  • Set daemonize to yes (by default it is set to no).

    # By default Redis does not run as a daemon. Use 'yes' if you need it.
    # Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
    
    daemonize yes
    

  • Set the pidfile to /var/run/redis_6379.pid (modify the port if needed).

    # If a pid file is specified, Redis writes it where specified at startup
    # and removes it at exit.
    #
    # When the server runs non daemonized, no pid file is created if none is
    # specified in the configuration. When the server is daemonized, the pid file
    # is used even if not specified, defaulting to "/var/run/redis.pid".
    #
    # Creating a pid file is best effort: if Redis is not able to create it
    # nothing bad happens, the server will start and run normally.
    
    pidfile /var/run/redis_6379.pid
    

  • Change the port accordingly. In our example it is not needed as the default port is already 6379.

    # Accept connections on the specified port, default is 6379 (IANA #815344).
    # If port 0 is specified Redis will not listen on a TCP socket.
    
    port 6379
    

  • Set your preferred loglevel.

    # Specify the server verbosity level.
    # This can be one of:
    # debug (a lot of information, useful for development/testing)
    # verbose (many rarely useful info, but not a mess like the debug level)
    # notice (moderately verbose, what you want in production probably)
    # warning (only very important / critical messages are logged)
    
    loglevel notice
    

  • Set the logfile to /var/log/redis_6379.log

    # Specify the log file name. Also the empty string can be used to force
    # Redis to log on the standard output. Note that if you use standard
    # output for logging but daemonize, logs will be sent to /dev/null
    
    logfile "/var/log/redis_6379.log"
    

  • Set the dir to /var/redis/6379

    # The working directory.
    #
    # The DB will be written inside this directory, with the filename specified
    # above using the 'dbfilename' configuration directive.
    #
    # The Append Only File will also be created inside this directory.
    #
    # Note that you must specify a directory here, not a file name.
    
    dir /var/redis/6379
    

  • Uncomment line # bind 127.0.0.1 (very important step for security reasons! With such settings redis will be accessible only from localhost. It will reject connections from outside network.)

    # ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
    # internet, binding to all the interfaces is dangerous and will expose the
    # instance to everybody on the internet. So by default we uncomment the
    # following bind directive, that will force Redis to listen only into
    # the IPv4 lookback interface address (this means Redis will be able to
    # accept connections only from clients running into the same computer it
    # is running).
    #
    # IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
    # JUST COMMENT THE FOLLOWING LINE.
    # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    bind 127.0.0.1
    

• Finally add the new Redis init script to all the default runlevels using the following command:

  sudo update-rc.d redis_6379 defaults
  

You are done! Now you can try running your instance with:

sudo /etc/init.d/redis_6379 start

Make sure that everything is working as expected:

• Try pinging your instance with redis-cli ping.
• Do a test save with redis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
• Check that your Redis instance is correctly logging in the log file /var/log/redis_6379.log.
• If it's a new machine where you can try it without problems make sure that after a reboot everything is still working.

Install MiaRec web application

1.1. Installed required packages

sudo apt-get install libpq-dev gcc libffi-dev libssl-dev python3-setuptools python3-dev

1.2. Create python virtual environment

It is best practice to install MiaRec web application into a "virtual" Python environment in order to obtain isolation from any "system" packages you've got installed in your Python version. This can be done by using the virtualenv package. Using a virtualenv will also prevent MiaRec from globally installing versions of packages that are not compatible with your system Python.

Install python pip package:

sudo apt-get install python3-pip

Install python venv package (modify python version in this command according to your python version, see output of python3 --version):

sudo apt-get install python3.4-venv

Create virtual environment:

sudo apt-get install python3-venv

sudo easy_install3 virtualenv

Create virtual environment:

mkdir /var/www/miarec
python3 -m venv /var/www/miarec/pyenv

1.3. Install MiaRec web application

Contact us (sales@miarec.com) to receive URL to MiaRec installation files.

Download MiaRec web application archive:

wget CONTACT_US_FOR_URL

Extract:

tar -xzvf miarecweb*.tar.gz

Move it to /var/www/miarec/app

mv miarecweb-*/ /var/www/miarec/app

Activate virtual environment:

sudo su
source /var/www/miarec/pyenv/bin/activate

Note, sudo su command is necessary because by default /var/www/ directory is not writable for non-root users in Ubuntu. We are going to install MiaRec web portal files into that directory. That's why you need to switch to root account now.

Upgrade pip to the latest version:

pip install --upgrade pip

Install MiaRec web application into python environment:

pip install -e /var/www/miarec/app

Create log and cache directories for MiaRec web application:

mkdir /var/log/miarecweb
mkdir /var/www/miarec/cache

Make Apache an owner of theses directory. So, it can create log and cache files there.

chown www-data:www-data /var/log/miarecweb
chown www-data:www-data /var/www/miarec/cache

2. Configure MiaRec web portal application

Copy production.ini file from a sample file:

cp /var/www/miarec/app/production.ini.sample /var/www/miarec/production.ini

Edit production.ini file:

vim /var/www/miarec/production.ini

Change in this file the following parameters according to previously installed PostgreSQL and Redis:

• DATABASE_HOST (use 127.0.0.1 if database is installed on the same host)
• DATABASE_PORT (default is 5432)
• DATABASE_NAME (should match to previously created database name, default is miarecdb)
• DATABASE_USER (should match to previously created database user for miarec, default is miarec)
• DATABASE_PASSWORD (should match to previously created database user password)
• REDIS_HOST (use 127.0.0.1 if Redis is installed on the same host)
• REDIS_PORT (default is 6379)

If Redis service is configured with non-default port (which is 6379), then replace 6379 with appropriate port number. If Redis service is running on a dedicated server, then replace 127.0.0.1 to appropriate ip-address.

3. Initialize MiaRec database layout

source /var/www/miarec/pyenv/bin/activate

alembic -c /var/www/miarec/production.ini upgrade head

4. Install Apache mod_wsgi module.

sudo apt-get install libapache2-mod-wsgi-py3

Copy miarec.wsgi.sample into miarec.wsgi

cp /var/www/miarec/app/miarec.wsgi.sample /var/www/miarec/miarec.wsgi

5. Edit Apache configuration file

Create miarec.conf file inside /etc/apache2/sites-available directory:

vi /etc/apache2/sites-available/miarec.conf

Content of this file:

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=www-data group=www-data python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec
WSGIProcessGroup miarec

<Directory /var/www/miarec/app>
  Require all granted
</Directory>

Disable a default site:

a2dissite 000-default

Enable miarec site:

a2ensite miarec

Restart Apache:

service apache2 reload

6. Access MiaRec web-portal with web-browser

Now you should be access MiaRec from web browser with URL http://

You may need to configure firewall exception rules on the server to allow inbound connections to the server on port 80.

Install Celery task manager

Celery is an asynchronous task queue/job queue system, which is used by MiaRec web portal for different tasks (long running tasks and/or periodic jobs).

Celery itself is already installed on your system when you deployed MiaRecWeb portal. The only missing part is to run Celery as a daemon.

There are two celery daemons:

• Celery worker executes long-running jobs like call backup/restore, purge deleted records etc.
• Celery scheduler manages periodic tasks. It loads job schedule configuration from MiaRec and initiates execution of these jobs by Celery worker at regular intervals.

The current document is based on official celery documentation.

1. Celery worker daemon (celeryd)

1.1. Create init.d startup script

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celeryd

Make it executable:

chmod +x celeryd

Move to /etc/init.d/

mv celeryd /etc/init.d/

1.2. Create celery worker configuration file

vi /etc/default/celeryd

Content of this file should be:

# Names of nodes to start
CELERYD_NODES="worker1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYD_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8 --ini-file=/var/www/miarec/production.ini"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/miarec/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYD_USER="root"
CELERYD_GROUP="root"

1.3. Install this init.d script and configure it to start automatically during boot process

update-rc.d celeryd defaults

1.4. Start celery

service celeryd start

2. Celery scheduler daemon (celerybeat)

2.1. Create init.d startup script for celery scheduler

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celerybeat

Make it executable:

chmod +x celerybeat

Move to /etc/init.d/

mv celerybeat /etc/init.d/

2.2. Create celery scheduler configuration file

vi /etc/default/celerybeat

Content of this file should be:

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYBEAT_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the scheduler
CELERYBEAT_OPTS="-S miarecweb.jobs.scheduler.JobScheduler --ini-file /var/www/miarec/production.ini"

CELERYBEAT_LOG_FILE="/var/log/miarec/celery/beat.log"
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYBEAT_USER="root"
CELERYBEAT_GROUP="root"

2.3. Install this init.d script and configure it to start automatically during boot process

update-rc.d celerybeat defaults

2.4. Start celery beat

service celerybeat start