Second Phase - GSoC, work on 3 VM setup of oVirt installation
30 Jul 2017 #oss #python #ansible #gsoc
It has been a week since the Phase 2 results are out. And we proceed to the last and final Phase of the GSoC. I couldn’t blog regularly in the last phase due to many reasons, which I want to change this Phase. But anyhow, this was collectively the output of my work in Phase 2
Setup
This approach will follow a 3 box VM setup.
For clarity sake, the VM’s can be assumed for now as
- VM A:
engine.ovirt.org: which storedenginedb and the main engine installation. - VM B:
dwhservice.ovirt.org: which will host the dwh service - VM C:
dwhdb.ovirt.org: which will store theovirt_engine_historydb
On VM A
---
- hosts: engine
vars:
ovirt_engine_type: 'ovirt-engine'
ovirt_engine_version: '4.1'
ovirt_engine_organization: 'dwhmanualenginetest.ovirt.org'
ovirt_engine_admin_password: 'secret'
ovirt_rpm_repo: 'http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'
ovirt_engine_organization: 'ovirt.org'
ovirt_engine_dwh_db_host: 'remotedwh.ovirt.org'
ovirt_engine_dwh_db_configure: false
roles:
- role: ovirt-common
- role: ovirt-engine-install-packages
- role: ovirt-engine-setup
Running this would be
$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
[WARNING]: While constructing a mapping from /Users/tasdikrahman/development/gsoc/ovirt-ansible/site.yml, line 4, column 5,
found a duplicate dict key (ovirt_engine_organization). Using last defined value only.
PLAY [engine] *****************************************************************************************************************
TASK [Gathering Facts] ********************************************************************************************************
ok: [engine.ovirt.org]
TASK [ovirt-common : complain if no ovirt source is specified] ****************************************************************
skipping: [engine.ovirt.org]
TASK [ovirt-common : install libselinux-python for ansible] *******************************************************************
ok: [engine.ovirt.org]
TASK [ovirt-common : creating directory repo-backup in yum.repos.d] ***********************************************************
changed: [engine.ovirt.org]
TASK [ovirt-common : create repository backup] ********************************************************************************
changed: [engine.ovirt.org]
TASK [ovirt-common : copy repository files] ***********************************************************************************
TASK [ovirt-common : install rpm repository package] **************************************************************************
changed: [engine.ovirt.org]
TASK [ovirt-common : create repository files] *********************************************************************************
TASK [ovirt-engine-install-packages : yum install engine] *********************************************************************
changed: [engine.ovirt.org]
TASK [ovirt-engine-setup : check if ovirt-engine running (health page)] *******************************************************
FAILED - RETRYING: check if ovirt-engine running (health page) (2 retries left).
FAILED - RETRYING: check if ovirt-engine running (health page) (1 retries left).
fatal: [engine.ovirt.org]: FAILED! => {"attempts": 2, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://engine.ovirt.org/ovirt-engine/services/health"}
...ignoring
TASK [ovirt-engine-setup : copy default answerfile] ***************************************************************************
changed: [engine.ovirt.org]
TASK [ovirt-engine-setup : copy custom answer file] ***************************************************************************
skipping: [engine.ovirt.org]
TASK [ovirt-engine-setup : update setup packages] *****************************************************************************
skipping: [engine.ovirt.org]
TASK [ovirt-engine-setup : run engine-setup with answerfile] ******************************************************************
changed: [engine.ovirt.org]
TASK [ovirt-engine-setup : check state of database] ***************************************************************************
ok: [engine.ovirt.org]
▽
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
TASK [ovirt-engine-setup : check state of engine] *****************************************************************************
ok: [engine.ovirt.org]
TASK [ovirt-engine-setup : restart of ovirt-engine service] *******************************************************************
changed: [engine.ovirt.org]
TASK [ovirt-engine-setup : Open port 5432 for opening connection to remote DWH if not being setup on this host] ***************
changed: [engine.ovirt.org] => (item=5432)
TASK [ovirt-engine-setup : check health status of page] ***********************************************************************
FAILED - RETRYING: check health status of page (12 retries left).
FAILED - RETRYING: check health status of page (11 retries left).
FAILED - RETRYING: check health status of page (10 retries left).
FAILED - RETRYING: check health status of page (9 retries left).
ok: [engine.ovirt.org]
TASK [ovirt-engine-setup : clean tmp files] ***********************************************************************************
changed: [engine.ovirt.org]
RUNNING HANDLER [ovirt-engine-setup : Reload firewalld] ***********************************************************************
changed: [engine.ovirt.org]
PLAY RECAP ********************************************************************************************************************
engine.ovirt.org : ok=16 changed=10 unreachable=0 failed=0
On VM C
On testdwhdb.ovirt.org
$ ssh root@dwhdb.ovirt.org
[root@dwhdb ~]# hostname
dwhdb.ovirt.org
[root@dwhdb ~]# yum install postgresql-server -y > /dev/null
[root@dwhdb ~]# su -l postgres -c "/usr/bin/initdb --locale=en_US.UTF8 --auth='ident' --pgdata=/var/lib/pgsql/data/"
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
Success. You can now start the database server using:
/usr/bin/postgres -D /var/lib/pgsql/data
or
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
[root@dwhdb ~]#
[root@dwhdb ~]# systemctl start postgresql.service
[root@dwhdb ~]# systemctl enable postgresql.service
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql.service to /usr/lib/systemd/system/postgresql.service.
[root@dwhdb ~]# su - postgres
-bash-4.2$ psql
psql (9.2.18)
Type "help" for help.
postgres=# create role ovirt_engine_history with login encrypted password 'password';
CREATE ROLE
postgres=# create database ovirt_engine_history owner ovirt_engine_history template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';
postgres=# \q
-bash-4.2$ exit
[root@dwhdb ~]# tail -3 /var/lib/pgsql/data/pg_hba.conf
host ovirt_engine_history ovirt_engine_history xxx.xx.3.0/24 md5 # engine
host ovirt_engine_history ovirt_engine_history xxx.xx.11.0/24 md5 # dwhservice
[root@dwhdb ~]#
[root@dwhdb ~]# sed -n '59,66p' /var/lib/pgsql/data/postgresql.conf
listen_addresses = '*' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
#port = 5432 # (change requires restart)
# Note: In RHEL/Fedora installations, you can't set the port number here;
# adjust it in the service file instead.
max_connections = 150 # (change requires restart)
[root@dwhdb ~]# systemctl start firewalld
[root@dwhdb ~]# firewall-cmd --zone=public --add-port=5432/tcp --permanent
success
[root@dwhdb ~]# firewall-cmd --reload
success
[root@dwhdb ~]# iptables -S | grep 5432
-A IN_public_allow -p tcp -m tcp --dport 5432 -m conntrack --ctstate NEW -j ACCEPT
[root@dwhdb ~]#
So now that we have both the engine and ovirt_engine_history in place. We need to set up the VM which will setup up the dwh service on it.
dwh service VM
$ ssh root@dwhservice.ovirt.org
[root@dwhservice ~]# hostname
dwhservice.ovirt.org
[root@dwhservice ~]# yum install -y http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm > /dev/null
[root@dwhservice ~]# yum install -y ovirt-engine-dwh-setup > /dev/null
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/7/x86_64/repodata/fe30bd7c1f6f8d6f4007e9096a0c0fdd305c550f8c136ba86793577edc9dc571-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article
https://access.redhat.com/articles/1320623
If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/
warning: /var/cache/yum/x86_64/7/ovirt-4.1-epel/packages/libtommath-0.42.0-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
warning: /var/cache/yum/x86_64/7/ovirt-4.1/packages/ovirt-engine-lib-4.1.4.2-1.el7.centos.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID fe590cb7: NOKEY
Importing GPG key 0xFE590CB7:
Userid : "oVirt <infra@ovirt.org>"
Fingerprint: 31a5 d783 7fad 7cb2 86cd 3469 ab8c 4f9d fe59 0cb7
Package : ovirt-release41-4.1.4-1.el7.centos.noarch (installed)
From : /etc/pki/rpm-gpg/RPM-GPG-ovirt-4.1
Importing GPG key 0x352C64E5:
Userid : "Fedora EPEL (7) <epel@fedoraproject.org>"
Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
[root@dwhservice ~]# engine-setup
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf']
Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
Version: otopi-1.6.2 (otopi-1.6.2-1.el7.centos)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment customization
--== PRODUCT OPTIONS ==--
Configure Data Warehouse on this host (Yes, No) [Yes]:
--== PACKAGES ==--
[ INFO ] Checking for product updates...
[ INFO ] No product updates found
--== NETWORK CONFIGURATION ==--
Host fully qualified DNS name of this server [dwhservice.ovirt.org]:
[WARNING] Failed to resolve dwhservice.ovirt.org using DNS, it can be resolved only locally
Setup can automatically configure the firewall on this system.
Note: automatic configuration of the firewall may overwrite current settings.
Do you want Setup to configure the firewall? (Yes, No) [Yes]:
The following firewall managers were detected on this system: firewalld
Firewall manager to configure (firewalld): firewalld
[ INFO ] firewalld will be configured as firewall manager.
Host fully qualified DNS name of the engine server []: engine.ovirt.org
Setup will need to do some actions on the remote engine server. Either automatically, using ssh as root to access it, or you will be prompted to manually perform each such action.
Please choose one of the following:
1 - Access remote engine server using ssh as root
2 - Perform each action manually, use files to copy content around
(1, 2) [1]:
ssh port on remote engine server [22]:
root password on remote engine server engine.ovirt.org:
--== DATABASE CONFIGURATION ==--
Where is the DWH database located? (Local, Remote) [Local]: Remote
ATTENTION
Manual action required.
Please create database for ovirt-engine use. Use the following commands as an example:
create role ovirt_engine_history with login encrypted password '<password>';
create database ovirt_engine_history owner ovirt_engine_history
template template0
encoding 'UTF8' lc_collate 'en_US.UTF-8'
lc_ctype 'en_US.UTF-8';
Make sure that database can be accessed remotely.
DWH database host [localhost]: dwhdb.ovirt.org
DWH database port [5432]:
DWH database secured connection (Yes, No) [No]:
DWH database name [ovirt_engine_history]:
DWH database user [ovirt_engine_history]:
DWH database password:
Please provide the following credentials for the Engine database.
They should be found on the Engine server in '/etc/ovirt-engine/engine.conf.d/10-setup-database.conf'.
Engine database host []: engine.ovirt.org
Engine database port [5432]:
Engine database secured connection (Yes, No) [No]:
Engine database name [engine]:
Engine database user [engine]:
Engine database password:
--== OVIRT ENGINE CONFIGURATION ==--
--== STORAGE CONFIGURATION ==--
--== PKI CONFIGURATION ==--
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
Please choose Data Warehouse sampling scale:
(1) Basic
(2) Full
(1, 2)[2]:
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation
--== CONFIGURATION PREVIEW ==--
Firewall manager : firewalld
Update Firewall : True
Host FQDN : dwhservice.ovirt.org
Engine database secured connection : False
Engine database user name : engine
Engine database name : engine
Engine database host : engine.ovirt.org
Engine database port : 5432
Engine database host name validation : False
DWH installation : True
DWH database secured connection : False
DWH database host : dwhdb.ovirt.org
DWH database user name : ovirt_engine_history
DWH database name : ovirt_engine_history
DWH database port : 5432
DWH database host name validation : False
Please confirm installation settings (OK, Cancel) [OK]:
[ INFO ] Stage: Transaction setup
[ INFO ] Stopping dwh service
[ INFO ] Stage: Misc configuration
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Creating/refreshing DWH database schema
[ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
--== SUMMARY ==--
[ INFO ] Starting dwh service
Please restart the engine by running the following on engine.ovirt.org :
# service ovirt-engine restart
This is required for the dashboard to work.
--== END OF SUMMARY ==--
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170802052510-setup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of setup completed successfully
[root@dwhservice ~]# service restart ovirt-engine-dwh
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
[root@dwhservice ~]#
[root@dwhservice ~]# service ovirt-engine-dwhd restart
Redirecting to /bin/systemctl restart ovirt-engine-dwhd.service
Now restart the service systemctl restart ovirt-engine on the testengine.ovirt.org VM and go to https://testengine.ovirt.org/ovirt-engine/
Automatinng it using Ansible
The architecture of the overall system is similar to the one above.
Setup the testengine.ovirt.org the way we did the setup in the first step.
Next would be setup the postgresql instance which would host our ovirt_engine_history, we can do
On VM C
the VM has to be setup for the ovirt_engine_history VM which is to be used by dwhservice.ovirt.org VM.
---
- hosts: dwhdb
vars:
# the below vars are explained in `install-postgresql/defaults/main.yml` and
# also the other configurable variables are placed there
engine_vm_network_cidr: '139.162.45.0/24' # Network where the Engine VM lies
dwhservice_vm_network_cidr: '139.162.61.0/24'
ovirt_engine_dwh_db_password: 'password'
roles:
- role: ovirt-engine-remote-dwh-setup/install-postgresql
Running this would be
$ ansible-playbook site.yml -i inventory
PLAY [dwhdb] ******************************************************************************************************************
TASK [Gathering Facts] ********************************************************************************************************
ok: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
fatal: [testdwhdb.ovirt.org]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service postgresql: host"}
...ignoring
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : install postgresql] **************************************************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Check if the db is initialized] **************************************
ok: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Initialize the postgresql db] ****************************************
[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running su
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Start postgresql.service] ********************************************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : creating directory for sql scripts in /tmp/ansible-sql] **************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : copy SQL scripts] ****************************************************
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-user-create.sql)
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-create.sql)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : create engine DWH DB and user] ***************************************
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-user-create.sql)
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-create.sql)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Adding engine and dwhservice vm IP's in the dwhdb conf to be acessed remotely] ***
changed: [testdwhdb.ovirt.org] => (item=host ovirt_engine_history ovirt_engine_history 139.162.45.0/24 md5 # engine)
changed: [testdwhdb.ovirt.org] => (item=host ovirt_engine_history ovirt_engine_history 139.162.61.0/24 md5 # dwhservice)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Edit the config file /var/lib/pgsql/data/postgresql.conf] ************
changed: [testdwhdb.ovirt.org] => (item=sed -i -- 's/max_connections = 100/max_connections = 150/g' /var/lib/pgsql/data/postgresql.conf)
changed: [testdwhdb.ovirt.org] => (item=sed -i "60ilisten_addresses = '*'" /var/lib/pgsql/data/postgresql.conf)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable firewalld and open up port 5432] ******************************
changed: [testdwhdb.ovirt.org] => (item=systemctl start firewalld)
changed: [testdwhdb.ovirt.org] => (item=firewall-cmd --zone=public --add-port=5432/tcp --permanent)
changed: [testdwhdb.ovirt.org] => (item=firewall-cmd --reload)
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Restart postgresql for the loading the newer configs] ****************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : clean tmp files] *****************************************************
changed: [testdwhdb.ovirt.org]
TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable postgresql.service to start at boot time] *********************
ok: [testdwhdb.ovirt.org]
PLAY RECAP ********************************************************************************************************************
testdwhdb.ovirt.org : ok=16 changed=12 unreachable=0 failed=0
On VM B
Now VM which hosts the dwhservice needs to be setup.
---
- hosts: dwhservice
vars:
ovirt_engine_type: 'ovirt-engine'
ovirt_engine_version: '4.1'
ovirt_rpm_repo: 'http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'
ovirt_engine_host_root_passwd: 'pycon2017' # the root password of the host where ovirt-engine is installed
ovirt_engine_firewall_manager: 'firewalld'
ovirt_engine_host_fqdn: 'testengine.ovirt.org' # FQDN of the ovirt-engine installation host, should be resolvable from the new DWH host
ovirt_engine_db_host: 'testengine.ovirt.org'
ovirt_engine_dwh_db_host: 'testdwhdb.ovirt.org'
ovirt_engine_dwh_db_password: 'password'
ovirt_engine_history_db_on_dwhservice_host: False
roles:
- role: ovirt-common
- role: ovirt-engine-install-packages
- role: ovirt-engine-remote-dwh-setup
Running this would be
$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine,skip_yum_install_ovirt_engine_dwh -vvv
Now restart ovirt-engine in VMA by doing a systemctl restart ovirt-engine
Issues Faced
The very first thing which I faced as an issue was that, there was no proper documentation in the oVirt Downstream docs in place (as on now) in place. This made me research more about the things. Good in one way, anyway I raised report on bugzilla about here. https://bugzilla.redhat.com/show_bug.cgi?id=1475706 There was another snag like the previous one where an OTOPI variable was not being logged in the answerfiles which led me to
Reference Commits
- Adds remote postgresql installation playbook
- Adds role for the dwh service setup along with the remaining three VMS’s
Future Work
- Cleanup of the PR’s opened so far.
- Work on test upgrade playbook.
- role for ovirt metrics deployment
- role for direct upgrade
References
- http://www.ovirt.org/documentation/install-guide/appe-Preparing_a_Remote_PostgreSQL_Database_for_Use_with_the_oVirt_Engine/
- https://www.linode.com/docs/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7
- https://www.postgresql.org/docs/8.1/static/sql-createrole.html
- https://support.rackspace.com/how-to/postgresql-creating-and-dropping-roles/
I have to say, I got stuck a lot while trying this out and there have numerous times where I was just about to give up. But my mentor was always there to support my questions and doubts. Thanks for that Lukas :)
More to come