Second Phase - GSoC, work on 3 VM setup of oVirt installation

Second Phase - GSoC, work on 3 VM setup of oVirt installation

It has been a week since the Phase 2 results are out. And we proceed to the last and final Phase of the GSoC. I couldn’t blog regularly in the last phase due to many reasons, which I want to change this Phase. But anyhow, this was collectively the output of my work in Phase 2


This approach will follow a 3 box VM setup.

For clarity sake, the VM’s can be assumed for now as

- hosts: engine
    ovirt_engine_type: 'ovirt-engine'
    ovirt_engine_version: '4.1'
    ovirt_engine_organization: ''
    ovirt_engine_admin_password: 'secret'
    ovirt_rpm_repo: ''
    ovirt_engine_organization: ''
    ovirt_engine_dwh_db_host: ''
    ovirt_engine_dwh_db_configure: false
    - role: ovirt-common
    - role: ovirt-engine-install-packages
    - role: ovirt-engine-setup

Running this would be

$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
 [WARNING]: While constructing a mapping from /Users/tasdikrahman/development/gsoc/ovirt-ansible/site.yml, line 4, column 5,
found a duplicate dict key (ovirt_engine_organization). Using last defined value only.

PLAY [engine] *****************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: []

TASK [ovirt-common : complain if no ovirt source is specified] ****************************************************************
skipping: []

TASK [ovirt-common : install libselinux-python for ansible] *******************************************************************
ok: []

TASK [ovirt-common : creating directory repo-backup in yum.repos.d] ***********************************************************
changed: []

TASK [ovirt-common : create repository backup] ********************************************************************************
changed: []

TASK [ovirt-common : copy repository files] ***********************************************************************************

TASK [ovirt-common : install rpm repository package] **************************************************************************
changed: []

TASK [ovirt-common : create repository files] *********************************************************************************

TASK [ovirt-engine-install-packages : yum install engine] *********************************************************************
changed: []

TASK [ovirt-engine-setup : check if ovirt-engine running (health page)] *******************************************************
FAILED - RETRYING: check if ovirt-engine running (health page) (2 retries left).
FAILED - RETRYING: check if ovirt-engine running (health page) (1 retries left).
fatal: []: FAILED! => {"attempts": 2, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": ""}

TASK [ovirt-engine-setup : copy default answerfile] ***************************************************************************
changed: []

TASK [ovirt-engine-setup : copy custom answer file] ***************************************************************************
skipping: []

TASK [ovirt-engine-setup : update setup packages] *****************************************************************************
skipping: []

TASK [ovirt-engine-setup : run engine-setup with answerfile] ******************************************************************
changed: []

TASK [ovirt-engine-setup : check state of database] ***************************************************************************
ok: []

▽   localhost localhost.localdomain localhost4 localhost4.localdomain4

TASK [ovirt-engine-setup : check state of engine] *****************************************************************************
ok: []

TASK [ovirt-engine-setup : restart of ovirt-engine service] *******************************************************************
changed: []

TASK [ovirt-engine-setup : Open port 5432 for opening connection to remote DWH if not being setup on this host] ***************
changed: [] => (item=5432)

TASK [ovirt-engine-setup : check health status of page] ***********************************************************************
FAILED - RETRYING: check health status of page (12 retries left).
FAILED - RETRYING: check health status of page (11 retries left).
FAILED - RETRYING: check health status of page (10 retries left).
FAILED - RETRYING: check health status of page (9 retries left).
ok: []

TASK [ovirt-engine-setup : clean tmp files] ***********************************************************************************
changed: []

RUNNING HANDLER [ovirt-engine-setup : Reload firewalld] ***********************************************************************
changed: []

PLAY RECAP ********************************************************************************************************************           : ok=16   changed=10   unreachable=0    failed=0


$ ssh
[root@dwhdb ~]# hostname
[root@dwhdb ~]# yum install postgresql-server -y > /dev/null
[root@dwhdb ~]# su -l postgres -c "/usr/bin/initdb --locale=en_US.UTF8 --auth='ident' --pgdata=/var/lib/pgsql/data/"
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
Success. You can now start the database server using:
    /usr/bin/postgres -D /var/lib/pgsql/data
    /usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
[root@dwhdb ~]#
[root@dwhdb ~]# systemctl start postgresql.service
[root@dwhdb ~]# systemctl enable postgresql.service
Created symlink from /etc/systemd/system/ to /usr/lib/systemd/system/postgresql.service.
[root@dwhdb ~]# su - postgres
-bash-4.2$ psql
psql (9.2.18)
Type "help" for help.
postgres=# create role ovirt_engine_history with login encrypted password 'password';
postgres=# create database ovirt_engine_history owner ovirt_engine_history template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';
postgres=# \q
-bash-4.2$ exit
[root@dwhdb ~]# tail -3 /var/lib/pgsql/data/pg_hba.conf
host    ovirt_engine_history    ovirt_engine_history    xxx.xx.3.0/24     md5 # engine
host    ovirt_engine_history    ovirt_engine_history    xxx.xx.11.0/24     md5 # dwhservice

[root@dwhdb ~]#
[root@dwhdb ~]# sed -n '59,66p' /var/lib/pgsql/data/postgresql.conf
listen_addresses = '*'		# what IP address(es) to listen on;
					# comma-separated list of addresses;
					# defaults to 'localhost'; use '*' for all
					# (change requires restart)
#port = 5432				# (change requires restart)
# Note: In RHEL/Fedora installations, you can't set the port number here;
# adjust it in the service file instead.
max_connections = 150			# (change requires restart)
[root@dwhdb ~]# systemctl start firewalld
[root@dwhdb ~]# firewall-cmd --zone=public --add-port=5432/tcp --permanent
[root@dwhdb ~]# firewall-cmd --reload
[root@dwhdb ~]# iptables -S | grep 5432
-A IN_public_allow -p tcp -m tcp --dport 5432 -m conntrack --ctstate NEW -j ACCEPT
[root@dwhdb ~]#

So now that we have both the engine and ovirt_engine_history in place. We need to set up the VM which will setup up the dwh service on it.

dwh service VM

$ ssh
[root@dwhservice ~]# hostname
[root@dwhservice ~]# yum install -y > /dev/null
[root@dwhservice ~]# yum install -y ovirt-engine-dwh-setup > /dev/null [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

If above article doesn't help to resolve this issue please create a bug on

warning: /var/cache/yum/x86_64/7/ovirt-4.1-epel/packages/libtommath-0.42.0-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
warning: /var/cache/yum/x86_64/7/ovirt-4.1/packages/ovirt-engine-lib- Header V4 RSA/SHA1 Signature, key ID fe590cb7: NOKEY
Importing GPG key 0xFE590CB7:
 Userid     : "oVirt <>"
 Fingerprint: 31a5 d783 7fad 7cb2 86cd 3469 ab8c 4f9d fe59 0cb7
 Package    : ovirt-release41-4.1.4-1.el7.centos.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-ovirt-4.1
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 From       :
[root@dwhservice ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf']
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
          Version: otopi-1.6.2 (otopi-1.6.2-1.el7.centos)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment customization

          --== PRODUCT OPTIONS ==--

          Configure Data Warehouse on this host (Yes, No) [Yes]:

          --== PACKAGES ==--

[ INFO  ] Checking for product updates...
[ INFO  ] No product updates found

          --== NETWORK CONFIGURATION ==--

          Host fully qualified DNS name of this server []:
[WARNING] Failed to resolve using DNS, it can be resolved only locally
          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]:
          The following firewall managers were detected on this system: firewalld
          Firewall manager to configure (firewalld): firewalld
[ INFO  ] firewalld will be configured as firewall manager.
          Host fully qualified DNS name of the engine server []:
          Setup will need to do some actions on the remote engine server. Either automatically, using ssh as root to access it, or you will be prompted to manually perform each such action.
          Please choose one of the following:
          1 - Access remote engine server using ssh as root
          2 - Perform each action manually, use files to copy content around
          (1, 2) [1]:
          ssh port on remote engine server [22]:
          root password on remote engine server

          --== DATABASE CONFIGURATION ==--

          Where is the DWH database located? (Local, Remote) [Local]: Remote


          Manual action required.
          Please create database for ovirt-engine use. Use the following commands as an example:

          create role ovirt_engine_history with login encrypted password '<password>';
          create database ovirt_engine_history owner ovirt_engine_history
           template template0
           encoding 'UTF8' lc_collate 'en_US.UTF-8'
           lc_ctype 'en_US.UTF-8';

          Make sure that database can be accessed remotely.

          DWH database host [localhost]:
          DWH database port [5432]:
          DWH database secured connection (Yes, No) [No]:
          DWH database name [ovirt_engine_history]:
          DWH database user [ovirt_engine_history]:
          DWH database password:

          Please provide the following credentials for the Engine database.
          They should be found on the Engine server in '/etc/ovirt-engine/engine.conf.d/10-setup-database.conf'.

          Engine database host []:
          Engine database port [5432]:
          Engine database secured connection (Yes, No) [No]:
          Engine database name [engine]:
          Engine database user [engine]:
          Engine database password:


          --== STORAGE CONFIGURATION ==--

          --== PKI CONFIGURATION ==--

          --== APACHE CONFIGURATION ==--

          --== SYSTEM CONFIGURATION ==--

          --== MISC CONFIGURATION ==--

          Please choose Data Warehouse sampling scale:
          (1) Basic
          (2) Full
          (1, 2)[2]:

          --== END OF CONFIGURATION ==--

[ INFO  ] Stage: Setup validation

          --== CONFIGURATION PREVIEW ==--

          Firewall manager                        : firewalld
          Update Firewall                         : True
          Host FQDN                               :
          Engine database secured connection      : False
          Engine database user name               : engine
          Engine database name                    : engine
          Engine database host                    :
          Engine database port                    : 5432
          Engine database host name validation    : False
          DWH installation                        : True
          DWH database secured connection         : False
          DWH database host                       :
          DWH database user name                  : ovirt_engine_history
          DWH database name                       : ovirt_engine_history
          DWH database port                       : 5432
          DWH database host name validation       : False

          Please confirm installation settings (OK, Cancel) [OK]:
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stopping dwh service
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up

          --== SUMMARY ==--

[ INFO  ] Starting dwh service
          Please restart the engine by running the following on :
          # service ovirt-engine restart
          This is required for the dashboard to work.

          --== END OF SUMMARY ==--

[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170802052510-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Execution of setup completed successfully
[root@dwhservice ~]# service restart ovirt-engine-dwh
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
[root@dwhservice ~]#
[root@dwhservice ~]# service ovirt-engine-dwhd restart
Redirecting to /bin/systemctl restart  ovirt-engine-dwhd.service

Now restart the service systemctl restart ovirt-engine on the VM and go to

Automatinng it using Ansible

The architecture of the overall system is similar to the one above.

Setup the the way we did the setup in the first step.

Next would be setup the postgresql instance which would host our ovirt_engine_history, we can do


the VM has to be setup for the ovirt_engine_history VM which is to be used by VM.

- hosts: dwhdb
    # the below vars are explained in `install-postgresql/defaults/main.yml` and 
    # also the other configurable variables are placed there
    engine_vm_network_cidr: '' # Network where the Engine VM lies
    dwhservice_vm_network_cidr: ''
    ovirt_engine_dwh_db_password: 'password'
    - role: ovirt-engine-remote-dwh-setup/install-postgresql

Running this would be

$ ansible-playbook site.yml -i inventory

PLAY [dwhdb] ******************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
fatal: []: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service postgresql: host"}

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : install postgresql] **************************************************
changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Check if the db is initialized] **************************************
ok: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Initialize the postgresql db] ****************************************
 [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running su

changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Start postgresql.service] ********************************************
changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : creating directory for sql scripts in /tmp/ansible-sql] **************
changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : copy SQL scripts] ****************************************************
changed: [] => (item=ovirt-engine-dwh-db-user-create.sql)
changed: [] => (item=ovirt-engine-dwh-db-create.sql)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : create engine DWH DB and user] ***************************************
changed: [] => (item=ovirt-engine-dwh-db-user-create.sql)
changed: [] => (item=ovirt-engine-dwh-db-create.sql)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Adding engine and dwhservice vm IP's in the dwhdb conf to be acessed remotely] ***
changed: [] => (item=host    ovirt_engine_history    ovirt_engine_history     md5 # engine)
changed: [] => (item=host    ovirt_engine_history    ovirt_engine_history     md5 # dwhservice)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Edit the config file /var/lib/pgsql/data/postgresql.conf] ************
changed: [] => (item=sed -i -- 's/max_connections = 100/max_connections = 150/g' /var/lib/pgsql/data/postgresql.conf)
changed: [] => (item=sed -i "60ilisten_addresses = '*'" /var/lib/pgsql/data/postgresql.conf)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable firewalld and open up port 5432] ******************************
changed: [] => (item=systemctl start firewalld)
changed: [] => (item=firewall-cmd --zone=public --add-port=5432/tcp --permanent)
changed: [] => (item=firewall-cmd --reload)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Restart postgresql for the loading the newer configs] ****************
changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : clean tmp files] *****************************************************
changed: []

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable postgresql.service to start at boot time] *********************
ok: []

PLAY RECAP ********************************************************************************************************************        : ok=16   changed=12   unreachable=0    failed=0

Now VM which hosts the dwhservice needs to be setup.

- hosts: dwhservice
    ovirt_engine_type: 'ovirt-engine'
    ovirt_engine_version: '4.1'
    ovirt_rpm_repo: ''
    ovirt_engine_host_root_passwd: 'pycon2017'  # the root password of the host where ovirt-engine is installed
    ovirt_engine_firewall_manager: 'firewalld'
    ovirt_engine_host_fqdn: ''  # FQDN of the ovirt-engine installation host, should be resolvable from the new DWH host
    ovirt_engine_db_host: '' 
    ovirt_engine_dwh_db_host: ''
    ovirt_engine_dwh_db_password: 'password'
    ovirt_engine_history_db_on_dwhservice_host: False
    - role: ovirt-common
    - role: ovirt-engine-install-packages
    - role: ovirt-engine-remote-dwh-setup

Running this would be

$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine,skip_yum_install_ovirt_engine_dwh -vvv

Now restart ovirt-engine in VMA by doing a systemctl restart ovirt-engine

Issues Faced

The very first thing which I faced as an issue was that, there was no proper documentation in the oVirt Downstream docs in place (as on now) in place. This made me research more about the things. Good in one way, anyway I raised report on bugzilla about here. There was another snag like the previous one where an OTOPI variable was not being logged in the answerfiles which led me to

Reference Commits

Future Work


I have to say, I got stuck a lot while trying this out and there have numerous times where I was just about to give up. But my mentor was always there to support my questions and doubts. Thanks for that Lukas :)

More to come