Installation on Linux (Ubuntu) manually (deprecated)

A manual installation is deprecated. Ansible-based installation is used now.

MiaRec solution consists of multiple components:

  • MiaRec Recorder
  • Database (PostgreSQL)
  • MiaRec Web portal

These components may be installed all on a single server or each on a dedicated server.

MiaRec Architecture

Install MiaRec Web portal

MiaRec web portal requires the following components:

  • PostgreSQL database
  • Apache web server
  • Python 3 (required for running web portal scripts)
  • Redis (high speed in-memory caching)
  • Celery (scheduler and background task manager)

See also: MiaRec Architecture

Install PostgreSQL

This guide provides instructions how to install PostgreSQL for MiaRec.

Preparing The System

Update system default applications:

sudo apt-get update

Configure PostgreSQL Apt repository

These instructions are based on http://www.postgresql.org/download/linux/ubuntu/

Postgres is included into default repository of Ubuntu LTS, but its version is not up to date.

Create the fileĀ /etc/apt/sources.list.d/pgdg.list and add a line for the repository:

deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main

Import the repository signing key, and update the package lists:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
sudo apt-key add -
sudo apt-get update

Then install the required version.

sudo apt-get install postgresql-9.4 postgresql-contrib-9.4

Create MiaRec user and database

The default database name and database user areĀ postgres. Switch to postgres user to perform the following operations.

In this example we create user 'miarec' and database 'miarecdb'.

First, start psql command-line interface as postgres user:

sudo -u postgres psql postgres

Create user for MiaRec application:

CREATE USER miarec PASSWORD 'password';

Create MiaRec database:

CREATE DATABASE miarecdb WITH ENCODING 'UNICODE' LC_COLLATE 'C' LC_CTYPE 'C' TEMPLATE template0;
ALTER DATABASE miarecdb OWNER TO miarec;

Connect to "miarecdb" database:

\c miarecdb;

Install uuid-ossp and hstore extensions into "miarecdb" database:

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "hstore";

Enter \q to exit from psql command-line interface:

\q

PostgreSQL Configuration

The postgresql server is using two main configuration files

  • /etc/postgresql/9.4/main/pg_hba.conf
  • /etc/postgresql/9.4/main/postgresql.conf

pg_hba.conf

Change authentication method from ident to md5 for localhost connections.

Change:

host    all   all     127.0.0.1/32       ident

To:

host    all   all     127.0.0.1/32        md5

When other MiaRec components are deployed on dedicated servers, then you need to add their ip-addresses to trust group. For example:

host    all   all     192.168.0.10/32    md5      # allow access from 192.168.0.10
host    all   all     192.168.1.1/24     md5      # allow access from network 192.168.1.1/24

postgresql.conf

If other MiaRec components (like recorder and web portal) are deployed on dedicated servers, then you need to configure postgres to accept network connections. Change:

listen_addresses = 'localhost'

to

listen_addresses = '*'

Restart PostgreSQL

service postgresql restart

Install Python

Install python 3 from default repository.

sudo apt-get install python3

Execute the following command and make sure that it is at lease version 3.4:

python3 --version

If it is older, then you need to install python 3 manually from sources.

Install Apache web server

Install Apache web server and required packages

sudo apt-get install apache2 apache2-dev openssl libssl-dev

Start Apache web server

sudo service apache2 start

Install Redis cache

Download Redis from http://redis.io/download:

wget http://download.redis.io/releases/redis-3.2.1.tar.gz

Extract it and compile with:

tar -xzvf redis-3.2.1.tar.gz
cd redis-3.2.1
make

Install binaries:

sudo make install

Create init script for redis

  • Create a directory where to store your Redis config files and your data:

    sudo mkdir /etc/redis
    sudo mkdir /var/redis
    
  • Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:

    sudo cp utils/redis_init_script /etc/init.d/redis_6379
    
  • Edit the init script.

    sudo vim /etc/init.d/redis_6379
    

    Add the following lines at the top of init script (below line #!/bin/sh):

    #!/bin/sh
    #
    # Simple Redis init.d script conceived to work on Linux systems
    # as it does use of the /proc filesystem.
    #
    # chkconfig:   - 85 15
    # description:  Redis is a persistent key-value database
    #
    # processname: redis
    ### BEGIN INIT INFO
    # Provides: redis_6379
    # Required-Start:    $network $remote_fs $local_fs
    # Required-Stop:     $network $remote_fs $local_fs
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: start and stop redis_6379
    # Description: Redis daemon
    ### END INIT INFO
    

    Make sure to modify REDIS_PORT accordingly to the port you are using. Both the pid file path and the configuration file name depend on the port number.

  • Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:

    sudo cp redis.conf /etc/redis/6379.conf
    
  • Create a directory inside /var/redis that will work as data and working directory for this Redis instance:

    sudo mkdir /var/redis/6379
    
  • Edit the configuration file, making sure to perform the following changes:

    sudo vim /etc/redis/6379.conf
    
    • Set daemonize to yes (by default it is set to no).

      # By default Redis does not run as a daemon. Use 'yes' if you need it.
      # Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
      
      daemonize yes
      
    • Set the pidfile to /var/run/redis_6379.pid (modify the port if needed).

      # If a pid file is specified, Redis writes it where specified at startup
      # and removes it at exit.
      #
      # When the server runs non daemonized, no pid file is created if none is
      # specified in the configuration. When the server is daemonized, the pid file
      # is used even if not specified, defaulting to "/var/run/redis.pid".
      #
      # Creating a pid file is best effort: if Redis is not able to create it
      # nothing bad happens, the server will start and run normally.
      
      pidfile /var/run/redis_6379.pid
      
    • Change the port accordingly. In our example it is not needed as the default port is already 6379.

      # Accept connections on the specified port, default is 6379 (IANA #815344).
      # If port 0 is specified Redis will not listen on a TCP socket.
      
      port 6379
      
    • Set your preferred loglevel.

      # Specify the server verbosity level.
      # This can be one of:
      # debug (a lot of information, useful for development/testing)
      # verbose (many rarely useful info, but not a mess like the debug level)
      # notice (moderately verbose, what you want in production probably)
      # warning (only very important / critical messages are logged)
      
      loglevel notice
      
    • Set the logfile to /var/log/redis_6379.log

      # Specify the log file name. Also the empty string can be used to force
      # Redis to log on the standard output. Note that if you use standard
      # output for logging but daemonize, logs will be sent to /dev/null
      
      logfile "/var/log/redis_6379.log"
      
    • Set the dir to /var/redis/6379

      # The working directory.
      #
      # The DB will be written inside this directory, with the filename specified
      # above using the 'dbfilename' configuration directive.
      #
      # The Append Only File will also be created inside this directory.
      #
      # Note that you must specify a directory here, not a file name.
      
      dir /var/redis/6379
      
    • Uncomment line # bind 127.0.0.1 (very important step for security reasons! With such settings redis will be accessible only from localhost. It will reject connections from outside network.)

      # ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
      # internet, binding to all the interfaces is dangerous and will expose the
      # instance to everybody on the internet. So by default we uncomment the
      # following bind directive, that will force Redis to listen only into
      # the IPv4 lookback interface address (this means Redis will be able to
      # accept connections only from clients running into the same computer it
      # is running).
      #
      # IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
      # JUST COMMENT THE FOLLOWING LINE.
      # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      
      bind 127.0.0.1
      
  • Finally add the new Redis init script to all the default runlevels using the following command:

    sudo update-rc.d redis_6379 defaults
    

You are done! Now you can try running your instance with:

sudo /etc/init.d/redis_6379 start

Make sure that everything is working as expected:

  • Try pinging your instance with redis-cli ping.
  • Do a test save with redis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
  • Check that your Redis instance is correctly logging in the log file /var/log/redis_6379.log.
  • If it's a new machine where you can try it without problems make sure that after a reboot everything is still working.

Install MiaRec web application

1.1. Installed required packages

sudo apt-get install libpq-dev gcc libffi-dev libssl-dev python3-setuptools python3-dev

1.2. Create python virtual environment

It is best practice to install MiaRec web application into a "virtual" Python environment in order to obtain isolation from any "system" packages you've got installed in your Python version. This can be done by using the virtualenv package. Using a virtualenv will also prevent MiaRec from globally installing versions of packages that are not compatible with your system Python.

Install python pip package:

sudo apt-get install python3-pip

Install python venv package (modify python version in this command according to your python version, see output of python3 --version):

sudo apt-get install python3.4-venv

Create virtual environment:

sudo apt-get install python3-venv

sudo easy_install3 virtualenv

Create virtual environment:

mkdir /var/www/miarec
python3 -m venv /var/www/miarec/pyenv

1.3. Install MiaRec web application

Contact us (sales@miarec.com) to receive URL to MiaRec installation files.

Download MiaRec web application archive:

wget CONTACT_US_FOR_URL

Extract:

tar -xzvf miarecweb*.tar.gz

Move it to /var/www/miarec/app

mv miarecweb-*/ /var/www/miarec/app

Activate virtual environment:

sudo su
source /var/www/miarec/pyenv/bin/activate

Note, sudo su command is necessary because by default /var/www/ directory is not writable for non-root users in Ubuntu. We are going to install MiaRec web portal files into that directory. That's why you need to switch to root account now.

Upgrade pip to the latest version:

pip install --upgrade pip

Install MiaRec web application into python environment:

pip install -e /var/www/miarec/app

Create log and cache directories for MiaRec web application:

mkdir /var/log/miarecweb
mkdir /var/www/miarec/cache

Make Apache an owner of theses directory. So, it can create log and cache files there.

chown www-data:www-data /var/log/miarecweb
chown www-data:www-data /var/www/miarec/cache

2. Configure MiaRec web portal application

Copy production.ini file from a sample file:

cp /var/www/miarec/app/production.ini.sample /var/www/miarec/production.ini

Edit production.ini file:

vim /var/www/miarec/production.ini

Change in this file the following parameters according to previously installed PostgreSQL and Redis:

  • DATABASE_HOST (use 127.0.0.1 if database is installed on the same host)
  • DATABASE_PORT (default is 5432)
  • DATABASE_NAME (should match to previously created database name, default is miarecdb)
  • DATABASE_USER (should match to previously created database user for miarec, default is miarec)
  • DATABASE_PASSWORD (should match to previously created database user password)
  • REDIS_HOST (use 127.0.0.1 if Redis is installed on the same host)
  • REDIS_PORT (default is 6379)

If Redis service is configured with non-default port (which is 6379), then replace 6379 with appropriate port number. If Redis service is running on a dedicated server, then replace 127.0.0.1 to appropriate ip-address.

3. Initialize MiaRec database layout

source /var/www/miarec/pyenv/bin/activate

alembic -c /var/www/miarec/production.ini upgrade head

4. Install Apache mod_wsgi module.

sudo apt-get install libapache2-mod-wsgi-py3

Copy miarec.wsgi.sample into miarec.wsgi

cp /var/www/miarec/app/miarec.wsgi.sample /var/www/miarec/miarec.wsgi

5. Edit Apache configuration file

Create miarec.conf file inside /etc/apache2/sites-available directory:

vi /etc/apache2/sites-available/miarec.conf

Content of this file:

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=www-data group=www-data python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec
WSGIProcessGroup miarec

<Directory /var/www/miarec/app>
  Require all granted
</Directory>

Disable a default site:

a2dissite 000-default

Enable miarec site:

a2ensite miarec

Restart Apache:

service apache2 reload

6. Access MiaRec web-portal with web-browser

Now you should be access MiaRec from web browser with URL http://

You may need to configure firewall exception rules on the server to allow inbound connections to the server on port 80.

Install Celery task manager

Celery is an asynchronous task queue/job queue system, which is used by MiaRec web portal for different tasks (long running tasks and/or periodic jobs).

Celery itself is already installed on your system when you deployed MiaRecWeb portal. The only missing part is to run Celery as a daemon.

There are two celery daemons:

  • Celery worker executes long-running jobs like call backup/restore, purge deleted records etc.
  • Celery scheduler manages periodic tasks. It loads job schedule configuration from MiaRec and initiates execution of these jobs by Celery worker at regular intervals.

The current document is based on official celery documentation.

1. Celery worker daemon (celeryd)

1.1. Create init.d startup script

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celeryd

Make it executable:

chmod +x celeryd

Move to /etc/init.d/

mv celeryd /etc/init.d/

1.2. Create celery worker configuration file

vi /etc/default/celeryd

Content of this file should be:

# Names of nodes to start
CELERYD_NODES="worker1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYD_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8 --ini-file=/var/www/miarec/production.ini"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/miarec/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYD_USER="root"
CELERYD_GROUP="root"

1.3. Install this init.d script and configure it to start automatically during boot process

update-rc.d celeryd defaults

1.4. Start celery

service celeryd start

2. Celery scheduler daemon (celerybeat)

2.1. Create init.d startup script for celery scheduler

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celerybeat

Make it executable:

chmod +x celerybeat

Move to /etc/init.d/

mv celerybeat /etc/init.d/

2.2. Create celery scheduler configuration file

vi /etc/default/celerybeat

Content of this file should be:

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYBEAT_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the scheduler
CELERYBEAT_OPTS="-S miarecweb.jobs.scheduler.JobScheduler --ini-file /var/www/miarec/production.ini"

CELERYBEAT_LOG_FILE="/var/log/miarec/celery/beat.log"
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYBEAT_USER="root"
CELERYBEAT_GROUP="root"

2.3. Install this init.d script and configure it to start automatically during boot process

update-rc.d celerybeat defaults

2.4. Start celery beat

service celerybeat start

Install MiaRec Recorder

1. Install required packages

sudo apt-get install libpcap

2. Download MiaRec installation files:

Contact us (sales@miarec.com) to receive URL to MiaRec installation files.

wget CONTACT_US_FOR_URL

Extract:

tar -xzvf miarec-*.tar.gz
cd miarec*

3. Install MiaRec recorder

Copy binary file to /usr/local/bin/

cp miarec /usr/local/bin/

Copy configuration files to /etc/miarec/

mkdir /etc/miarec
cp miarec.ini /etc/miarec/
cp -r sqlconfig /etc/miarec/

Create /var/lib/miarec directory. It will be used as current directory when running MiaRec process. MiaRec process reads SOAP wsdl file from current directory and stores some temporary files there.

mkdir /var/lib/miarec
cp WebServices.wsdl /var/lib/miarec/

Create log directories

mkdir /var/log/miarec

Create directory for recording files

mkdir -p /var/miarec/recordings

4. Create startup script (Upstart)

Create file /etc/init/miarec.conf

vi /etc/init/miarec.conf

Content of this file:

description "MiaRec call recorder"
author      "MiaRec, Inc. www.miarec.com"

env EXEC=/usr/local/bin/miarec
env PIDFILE=/var/run/miarec.pid
env CONFFILE=/etc/miarec/miarec.ini

start on started networking
stop on runlevel [!2345]

console output

# Restart automatically proces in case of crash
respawn

# Stop respawn if it occured more than 10 times during 60 seconds period.
# This means serious problems
respawn limit 10 60

# Current working directory for MiaRec process
chdir /var/lib/miarec

# Enable core dumps for troubleshooting
limit core unlimited unlimited

instance miarec
exec $EXEC -c $CONFFILE --pid $PIDFILE

Reload Upstart configuration

initctl reload-configuration

Validate that miarec is in a list of processes:

initctl list | grep miarec

If you do not see there miarec then check errors in /var/log/messages

Start MiaRec process

initctl start miarec

5. Edit miarec.ini configuration file

vi /etc/miarec/miarec.ini

Change database connection settings (host, port, database, user, password). There are two places in INI files, where you need to edit database settings:

5.1. Module which loads configuration from database

#-----------------------------------------------------------------
#    SQLConfig
#-----------------------------------------------------------------
#  Loading configuration from SQL database
#-----------------------------------------------------------------
################################################################################
[SQLConfig]
################################################################################

#  Database Driver type.
#  Supported values: 
#    PostgreSQL
#-------------------------------------------------------------------------------
Driver=PostgreSQL
#-------------------------------------------------------------------------------


#  Host of database server
#-------------------------------------------------------------------------------
Host=127.0.0.1:5432
#-------------------------------------------------------------------------------


#  Database name
#-------------------------------------------------------------------------------
Database=miarecdb
#-------------------------------------------------------------------------------


#  Username and password for accessing database. Should have write permissions.
#-------------------------------------------------------------------------------
Username=miarec
Password=password
#-------------------------------------------------------------------------------

5.2. Module, which writes call detail records (CDRs) to database:

#-----------------------------------------------------------------
#  Configuration section for SQLCallsLog module. This module stores calls log into database
#  Supported call events:
#    start,connect,update,stop,stream_start,stream_stop
#-----------------------------------------------------------------
################################################################################
[SQLCallsLog]
################################################################################

#  Database Driver type.
#  Supported values: 
#    PostgreSQL
#-------------------------------------------------------------------------------
Driver=PostgreSQL
#-------------------------------------------------------------------------------


#  Host of database server
#-------------------------------------------------------------------------------
Host=127.0.0.1:5432
#-------------------------------------------------------------------------------


#  Database name
#-------------------------------------------------------------------------------
Database=miarecdb
#-------------------------------------------------------------------------------


#  Username and password for accessing database. Should have write permissions.
#-------------------------------------------------------------------------------
Username=miarec
Password=password
#-------------------------------------------------------------------------------

6. Restart MiaRec service

initctl stop miarec
initctl start miarec

Configure firewall

By default MiaRec uses the following ports, which should be added into firewall exclusion list.

Port Description
80 (tcp) MiaRec Web-portal (HTTP protocol)
6554 (tcp) Live monitoring signaling (RTSP protocol)
7000 - 7999 (udp) Live monitoring media (RTP protocol)
5070 (tcp) Cisco SIP trunk recording signaling (SIP protocol)
20000 - 21999 (udp) Cisco SIP trunk recording media (RTP protocol)
5080 (tcp) SIPREC recording signaling (SIP protocol)
22000 - 23999 (udp) SIPREC recording media (RTP protocol)

This document describes how to configure iptables.

Execute command iptables --line -vnL to see the current list of rule with line numbers. Example output:

[root@miarec ~]# iptables --line -vnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     3124 1264K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
2        0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
3       11  3292 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
4        0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
5       63  4881 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT 2937 packets, 1212K bytes)
num   pkts bytes target     prot opt in     out     source               destination         

From this output we need to get the line number of the generic REJECT rule. In example above it is at line #5. We will need to add our exclusion rules just above this line.

Web-portal rule (port 80 tcp)

iptables -I INPUT 5 -i eth0 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT

Live monitoring rules

iptables -I INPUT 5 -i eth0 -p tcp --dport 6554 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT 5 -i eth0 -p udp --dport 7000:7999 -m state --state NEW,ESTABLISHED -j ACCEPT

Cisco SIP trunk recording interface rules

iptables -I INPUT 5 -i eth0 -p udp --dport 5070 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT 5 -i eth0 -p tcp --dport 5070 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT 5 -i eth0 -p udp --dport 20000:21999 -m state --state NEW,ESTABLISHED -j ACCEPT

SIPREC recording interface rules

iptables -I INPUT 5 -i eth0 -p udp --dport 5080 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT 5 -i eth0 -p tcp --dport 5080 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT 5 -i eth0 -p udp --dport 22000:23999 -m state --state NEW,ESTABLISHED -j ACCEPT

Save all rules into iptables configuration file

service iptables save

Restart iptables service

service iptables restart