Install MiaRec Web Portal

Please note: this is legacy documentation. Please check out https://docs.miarec.com/all/ for the most up-to-date documentation and user guides.

MiaRec web portal requires the following components:

PostgreSQL database
Apache web server
Python 3 (required for running web portal scripts)
Redis (high speed in-memory caching)
Celery (scheduler and background task manager)

Install PostgreSQL

This guide provides instructions on how to install PostgreSQL for MiaRec.

Preparing The System

Update system default applications:

yum update

Configure your YUM repository

Postgres is included in the default repository of CentOS / Red Hat / Fedora, but its version is not up to date.

Locate and edit your distributions .repo file, located:

On Fedora: /etc/yum.repos.d/fedora.repo and /etc/yum.repos.d/fedora-updates.repo, [fedora] sections
On CentOS: /etc/yum.repos.d/CentOS-Base.repo, [base] and [updates] sections
On Red Hat: /etc/yum/pluginconf.d/rhnplugin.conf [main] section

To the section(s) identified above, you need to append a line (otherwise dependencies might resolve to the PostgreSQL supplied by the base repository):

exclude=postgresql*

For example, on Centos 6 the .repo file should be:

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
exclude=postgresql*

#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
exclude=postgresql*

Install PGDG RPM file

A PGDG file is available for each distribution/architecture/database version combination. Browse http://yum.postgresql.org and find your correct RPM. For example, to install PostgreSQL 9.4 on CentOS 6 64-bit:

Centos 6:

yum localinstall http://yum.postgresql.org/9.4/redhat/rhel-6-x86_64/pgdg-centos94-9.4-2.noarch.rpm

Centos 7:

yum localinstall http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-2.noarch.rpm

Install PostgreSQL Database Server & Its Extensions

yum install postgresql94-server postgresql94-contrib

Initialize Data Directory

After installing the packages, a database needs to be initialized and configured.

service postgresql-9.4 initdb

If the previous command did not work, try directly calling the setup binary, located in:

/usr/pgsql-9.4/bin/postgresql94-setup initdb

Configure start at boot up

chkconfig postgresql-9.4 on

Start PostgreSQL database service

service postgresql-9.4 start

Create MiaRec user and database

In this example, we create user 'miarec' and database 'miarecdb'.

First, connect/login as root:

sudo -u postgres psql postgres

This will open psql command-line interface.

Create a user (replace 'password' with something more secure):

CREATE USER miarec PASSWORD 'password';

Create database:

CREATE DATABASE miarecdb WITH ENCODING 'UNICODE' LC_COLLATE 'C' LC_CTYPE 'C' TEMPLATE template0;
ALTER DATABASE miarecdb OWNER TO miarec;

Connect to "miarecdb" database:

\c miarecdb;

Install uuid-ossp and hstore extensions into "miarecdb" database:

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "hstore";

Enter \q to exit from psql command-line interface:

\q

PostgreSQL Configuration

The postgresql server is using two main configuration files

/var/lib/pgsql/9.4/data/pg_hba.conf
/var/lib/pgsql/9.4/data/postgresql.conf

Edit pg_hba.conf

vi /var/lib/pgsql/9.4/data/pg_hba.conf

Change authentication method from ident to md5 for localhost connections.

Change:

host    all   all     127.0.0.1/32       ident

To:

host    all   all     127.0.0.1/32        md5

When other MiaRec components are deployed on dedicated servers, then you need to add their ip-addresses to trust group. For example:

host    all   all     192.168.0.10/32    md5      # allow access from 192.168.0.10
host    all   all     192.168.1.1/24     md5      # allow access from network 192.168.1.1/24

Edit postgresql.conf (optional)

vi /var/lib/pgsql/9.4/data/postgresql.conf

If other MiaRec components (like recorder and web portal) are deployed on dedicated servers, then you need to configure postgres to accept inbound network connections. Change:

listen_addresses = 'localhost'

listen_addresses = '*'

Restart PostgreSQL

service postgresql-9.4 restart

Install Python 3

1. Preparing The System

Update system default applications:

yum update

Install required packages for building python:

yum install openssl-devel sqlite-devel bzip2-devel gcc wget

2. Build python 3 from source code

Download latest stable Python 3 source code files from https://www.python.org/downloads/source/. MiaRecWeb was tested with Python v.3.4

wget https://www.python.org/ftp/python/3.4.3/Python-3.4.3.tgz

Extract source code:

tar -xzvf Python-3.*.tgz

Build Python binaries:

cd Python-3*
./configure --enable-shared
make

Option --enable-shared is necessary in order to create shared library *.so files, which will be used later when compiling Apache mod_wsgi module.

3. Install Python

Normally, one would use “make install”; however, in order not to override system defaults - replacing the Python already used by the system - we will use make altinstall.

make altinstall

This will install python into /usr/local/bin with name, which contains version, like /usr/local/bin/python3.4.

4. Add Python shared libraries to PATH

By default Python installed *.so files into /usr/local/lib On some RedHat-based systems this directory is not searched when loading shared libraries. To fix that, you need to edit file /etc/ld.so.conf.

vi /etc/ld.so.conf

Add the following line to that file:

/usr/local/lib

Run ldconfig:

ldconfig

Install Apache Web Server

Install Apache web server and required packages:

yum install httpd httpd-devel openssl openssl-devel

Configure start at boot up:

chkconfig httpd on

Start Apache web server:

service httpd start

Install Redis Cache

Download Redis from http://redis.io/download:

wget http://download.redis.io/releases/redis-3.0.0.tar.gz

Extract it and compile with:

tar -xzvf redis-3.0.0.tar.gz
cd redis-3.0.0
make

Install binaries:

make install

Create a data directory for redis

This directory will work as data and working directory for your Redis instance:

mkdir -p /var/redis/6379

Create configuration file for redis

Create a directory where to store your Redis config files:

mkdir /etc/redis

Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:

cp redis.conf /etc/redis/6379.conf

Edit this file, making sure to perform the following changes:

vi /etc/redis/6379.conf

Set daemonize to yes (by default it is set to no).
Set the pidfile to /var/run/redis_6379.pid (modify the port if needed).
Change the port accordingly. In our example it is not needed as the default port is already 6379.
Set your preferred loglevel.
Set the logfile to /var/log/redis_6379.log
Set the dir to /var/redis/6379 (very important step!)
Uncomment line # bind 127.0.0.1 (very important step for security reasons! With such settings redis will be accessible only from localhost. It will reject connections from outside network.)

Create init script for redis

Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:
```
cp utils/redis_init_script /etc/init.d/redis_6379
```
Edit the init script.
```
vi /etc/init.d/redis_6379
```
Add the following lines at the top of init script (below line #!/bin/sh):
```
#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
#
# chkconfig:   - 85 15
# description:  Redis is a persistent key-value database
# processname: redis
```
Make sure to modify REDIS_PORT accordingly to the port you are using. Both the pid file path and the configuration file name depend on the port number.
Finally add the new Redis init script to all the default runlevels using the following command:
```
chkconfig --add redis_6379
chkconfig redis_6379 on
```

You are done! Now you can try running your instance with:

/etc/init.d/redis_6379 start

Verify Redis installation

Make sure that everything is working as expected:

Try pinging your instance with redis-cli ping.
Do a test save with redis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
Check that your Redis instance is correctly logging in the log file /var/log/redis_6379.log.
If it's a new machine where you can try it without problems make sure that after a reboot everything is still working.

Install MiaRec Web Application

1.1. Installed required packages

yum install postgresql94-devel gcc libffi-devel openssl-devel libxml2-devel libxslt-devel

1.2. Create python virtual environment

It is best practice to install MiaRec web application into a "virtual" Python environment in order to obtain isolation from any "system" packages you've got installed in your Python version. This can be done by using the virtualenv package. Using a virtualenv will also prevent MiaRec from globally installing versions of packages that are not compatible with your system Python.

Upgrade pip and setuptools to the latest version:

python3 -m pip install -U pip setuptools

On Centos 7 you may need to run python3.4 ... instead of python3 ...

Create virtual environment:

mkdir /var/www/miarec
python3 -m venv /var/www/miarec/pyenv

1.3. Install MiaRec web application

Fill the download form to request URL to MiaRec Web portal installation files.

Download MiaRec web application archive:

wget CONTACT_US_FOR_URL

Extract:

tar -xzvf miarecweb*.tar.gz

Move it to /var/www/miarec/app

mv miarecweb-*/ /var/www/miarec/app

Make sure that postgresql pg_config file is in PATH (it is required by psycopg2 python package). Execute in shell:

PATH=$PATH:/usr/pgsql-9.4/bin/

Activate virtual environment:

source /var/www/miarec/pyenv/bin/activate

Upgrade pip to the latest version:

pip install --upgrade pip

Install MiaRec web application into python environment:

pip install -e /var/www/miarec/app

Create log and cache directories for MiaRec web application:

mkdir /var/log/miarecweb
mkdir /var/www/miarec/cache

Make Apache an owner of these directories. So, it can create log and cache files there.

chown apache:apache /var/log/miarecweb
chown apache:apache /var/www/miarec/cache

2. Configure MiaRec web portal application

Create production.ini file from a sample file (production.ini.sample):

cp /var/www/miarec/app/production.ini.sample /var/www/miarec/production.ini

Edit production.ini file:

vi /var/www/miarec/production.ini

Change in this file the following parameters according to previously installed PostgreSQL and Redis:

DATABASE_HOST (use 127.0.0.1 if database is installed on the same host)
DATABASE_PORT (default is 5432)
DATABASE_NAME (should match to previously created database name, default is miarecdb)
DATABASE_USER (should match to previously created database user for miarec, default is miarec)
DATABASE_PASSWORD (should match to previously created database user password)
REDIS_HOST (use 127.0.0.1 if Redis is installed on the same host)
REDIS_PORT (default is 6379)

If Redis service is configured with non-default port (which is 6379), then replace 6379 with appropriate port number. If Redis service is running on a dedicated server, then replace 127.0.0.1 to appropriate ip-address.

3. Initialize MiaRec database layout

source /var/www/miarec/pyenv/bin/activate

alembic -c /var/www/miarec/production.ini upgrade head

4. Build and install Apache mod_wsgi module.

Download source code of mod_wsgi module from: https://github.com/GrahamDumpleton/mod_wsgi/releases

cd ~
wget https://github.com/GrahamDumpleton/mod_wsgi/archive/4.4.11.tar.gz

Activate previously created python virtual environment. This is necessary because CentOS has older Python version 2.7 and we need to build mod_wsgi for new python 3.4.

source /var/www/miarec/pyenv/bin/activate

Extract, build and install:

tar -xzvf 4.4.11.tar.gz
cd mod_wsgi*
./configure
make
make install

Disable SELinux because it causes Segmentation fault when Apache loads mod_wsgi module.

sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config

Reboot of server is required.

shutdown -r now

5. Edit Apache configuration file

Copy miarec.wsgi.sample into miarec.wsgi. This is a configuration for MiaRec WSGI application, which will be executed inside Apache web server.

cp /var/www/miarec/app/miarec.wsgi.sample /var/www/miarec/miarec.wsgi

Create miarec.conf file inside /etc/httpd/conf.d directory:

vi /etc/httpd/conf.d/miarec.conf

Content of this file:

Apache version 2.2 (for Centos 6):

LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix run/wsgi

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=apache group=apache python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIProcessGroup miarec

Alias /favicon.ico /var/www/miarec/app/miarecweb/static/favicon.ico
Alias /static /var/www/miarec/app/miarecweb/static

WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec

<Directory /var/www/miarec/app/miarecweb/static>
  AllowOverride None
  Order deny,allow
  Allow from all
</Directory>

<Directory /var/www/miarec>
  <Files miarec.wsgi>
      Order deny,allow
      Allow from all
  </Files>
</Directory>

Apache version 2.4 (for Centos 7):

LoadModule wsgi_module modules/mod_wsgi.so

WSGISocketPrefix run/wsgi

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.  See http://stackoverflow.com/a/10558360/209039
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On

WSGIDaemonProcess miarec user=apache group=apache python-path=/var/www/miarec/pyenv/lib/python3.4/site-packages
WSGIProcessGroup miarec

Alias /favicon.ico /var/www/miarec/app/miarecweb/static/favicon.ico
Alias /static /var/www/miarec/app/miarecweb/static

WSGIScriptAlias / /var/www/miarec/miarec.wsgi process-group=miarec

<Directory /var/www/miarec/app/miarecweb/static>
  AllowOverride None
  Require all granted
</Directory>

<Directory /var/www/miarec>
  <Files miarec.wsgi>
    Require all granted
  </Files>
</Directory>

Restart Apache:

service httpd restart

6. Access MiaRec web-portal with web-browser

Now you should be access MiaRec from web browser with URL http://ADDRESS

You may need to configure firewall exception rules on the server to allow inbound connections to the server on port 80.

7. Troubleshooting

If you receive "500 Internal Server Error" when accessing MiaRec web portal, then you can check Apache error log file:

less /var/log/httpd/error_log

Install Celery Task Manager

Celery is an asynchronous task queue/job queue system, which is used by MiaRec web portal for different tasks (long running tasks and/or periodic jobs).

Celery itself is already installed on your system when you deployed MiaRecWeb portal. The only missing part is to run Celery as a daemon.

There are two celery daemons:

Celery worker executes long-running jobs like call backup/restore, purge deleted records etc.
Celery scheduler manages periodic tasks. It loads job schedule configuration from MiaRec and initiates execution of these jobs by Celery worker at regular intervals.

The current document is based on official celery documentation.

1. Celery worker daemon (celeryd)

1.1. Create init.d startup script

Download default init.d script from celery Github repository:

cd ~
wget https://raw.githubusercontent.com/celery/celery/3.1/extra/generic-init.d/celeryd

Make it executable:

chmod +x celeryd

Move to /etc/init.d/

mv celeryd /etc/init.d/

1.2. Create celery worker configuration file

vi /etc/default/celeryd

Content of this file should be:

# Names of nodes to start
CELERYD_NODES="worker1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/miarec/pyenv/bin/celery"

# App instance to use
CELERY_APP="miarecweb.celery_app"

# Where to chdir at start.
CELERYD_CHDIR="/var/www/miarec/pyenv"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 -Ofair --concurrency=8 --ini-file=/var/www/miarec/production.ini"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/miarec/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Create log/pid dirs, if they don't already exist
CELERY_CREATE_DIRS=1

CELERYD_LOG_LEVEL="WARN"

CELERYD_USER="root"
CELERYD_GROUP="root"

1.3. Install this init.d script and configure it to start automatically during boot process

chkconfig --add celeryd
chkconfig celeryd on

1.5. Start celery

service celeryd start

2. Celery scheduler daemon (celerybeat)

2.1. Create init.d startup script for celery scheduler