Second Phase - GSoC, work on 3 VM setup of oVirt installation

Second Phase - GSoC, work on 3 VM setup of oVirt installation

It has been a week since the Phase 2 results are out. And we proceed to the last and final Phase of the GSoC. I couldn’t blog regularly in the last phase due to many reasons, which I want to change this Phase. But anyhow, this was collectively the output of my work in Phase 2

Setup

This approach will follow a 3 box VM setup.

For clarity sake, the VM’s can be assumed for now as

On VM A
---
- hosts: engine
  vars:
    ovirt_engine_type: 'ovirt-engine'
    ovirt_engine_version: '4.1'
    ovirt_engine_organization: 'dwhmanualenginetest.ovirt.org'
    ovirt_engine_admin_password: 'secret'
    ovirt_rpm_repo: 'http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'
    ovirt_engine_organization: 'ovirt.org'
    ovirt_engine_dwh_db_host: 'remotedwh.ovirt.org'
    ovirt_engine_dwh_db_configure: false
  roles:
    - role: ovirt-common
    - role: ovirt-engine-install-packages
    - role: ovirt-engine-setup

Running this would be

$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine_dwh,skip_yum_install_ovirt_engine_dwh_setup
 [WARNING]: While constructing a mapping from /Users/tasdikrahman/development/gsoc/ovirt-ansible/site.yml, line 4, column 5,
found a duplicate dict key (ovirt_engine_organization). Using last defined value only.


PLAY [engine] *****************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: [engine.ovirt.org]

TASK [ovirt-common : complain if no ovirt source is specified] ****************************************************************
skipping: [engine.ovirt.org]

TASK [ovirt-common : install libselinux-python for ansible] *******************************************************************
ok: [engine.ovirt.org]

TASK [ovirt-common : creating directory repo-backup in yum.repos.d] ***********************************************************
changed: [engine.ovirt.org]

TASK [ovirt-common : create repository backup] ********************************************************************************
changed: [engine.ovirt.org]

TASK [ovirt-common : copy repository files] ***********************************************************************************

TASK [ovirt-common : install rpm repository package] **************************************************************************
changed: [engine.ovirt.org]

TASK [ovirt-common : create repository files] *********************************************************************************

TASK [ovirt-engine-install-packages : yum install engine] *********************************************************************
changed: [engine.ovirt.org]

TASK [ovirt-engine-setup : check if ovirt-engine running (health page)] *******************************************************
FAILED - RETRYING: check if ovirt-engine running (health page) (2 retries left).
FAILED - RETRYING: check if ovirt-engine running (health page) (1 retries left).
fatal: [engine.ovirt.org]: FAILED! => {"attempts": 2, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://engine.ovirt.org/ovirt-engine/services/health"}
...ignoring

TASK [ovirt-engine-setup : copy default answerfile] ***************************************************************************
changed: [engine.ovirt.org]

TASK [ovirt-engine-setup : copy custom answer file] ***************************************************************************
skipping: [engine.ovirt.org]

TASK [ovirt-engine-setup : update setup packages] *****************************************************************************
skipping: [engine.ovirt.org]

TASK [ovirt-engine-setup : run engine-setup with answerfile] ******************************************************************
changed: [engine.ovirt.org]

TASK [ovirt-engine-setup : check state of database] ***************************************************************************
ok: [engine.ovirt.org]

▽
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

TASK [ovirt-engine-setup : check state of engine] *****************************************************************************
ok: [engine.ovirt.org]

TASK [ovirt-engine-setup : restart of ovirt-engine service] *******************************************************************
changed: [engine.ovirt.org]

TASK [ovirt-engine-setup : Open port 5432 for opening connection to remote DWH if not being setup on this host] ***************
changed: [engine.ovirt.org] => (item=5432)

TASK [ovirt-engine-setup : check health status of page] ***********************************************************************
FAILED - RETRYING: check health status of page (12 retries left).
FAILED - RETRYING: check health status of page (11 retries left).
FAILED - RETRYING: check health status of page (10 retries left).
FAILED - RETRYING: check health status of page (9 retries left).
ok: [engine.ovirt.org]

TASK [ovirt-engine-setup : clean tmp files] ***********************************************************************************
changed: [engine.ovirt.org]

RUNNING HANDLER [ovirt-engine-setup : Reload firewalld] ***********************************************************************
changed: [engine.ovirt.org]

PLAY RECAP ********************************************************************************************************************
engine.ovirt.org           : ok=16   changed=10   unreachable=0    failed=0
On VM C

On testdwhdb.ovirt.org

$ ssh [email protected]
[[email protected] ~]# hostname
dwhdb.ovirt.org
[[email protected] ~]# yum install postgresql-server -y > /dev/null
[[email protected] ~]# su -l postgres -c "/usr/bin/initdb --locale=en_US.UTF8 --auth='ident' --pgdata=/var/lib/pgsql/data/"
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/lib/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
Success. You can now start the database server using:
    /usr/bin/postgres -D /var/lib/pgsql/data
or
    /usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
[[email protected] ~]#
[[email protected] ~]# systemctl start postgresql.service
[[email protected] ~]# systemctl enable postgresql.service
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql.service to /usr/lib/systemd/system/postgresql.service.
[[email protected] ~]# su - postgres
-bash-4.2$ psql
psql (9.2.18)
Type "help" for help.
postgres=# create role ovirt_engine_history with login encrypted password 'password';
CREATE ROLE
postgres=# create database ovirt_engine_history owner ovirt_engine_history template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8' lc_ctype 'en_US.UTF-8';
postgres=# \q
-bash-4.2$ exit
[[email protected] ~]# tail -3 /var/lib/pgsql/data/pg_hba.conf
host    ovirt_engine_history    ovirt_engine_history    xxx.xx.3.0/24     md5 # engine
host    ovirt_engine_history    ovirt_engine_history    xxx.xx.11.0/24     md5 # dwhservice

[[email protected] ~]#
[[email protected] ~]# sed -n '59,66p' /var/lib/pgsql/data/postgresql.conf
listen_addresses = '*'		# what IP address(es) to listen on;
					# comma-separated list of addresses;
					# defaults to 'localhost'; use '*' for all
					# (change requires restart)
#port = 5432				# (change requires restart)
# Note: In RHEL/Fedora installations, you can't set the port number here;
# adjust it in the service file instead.
max_connections = 150			# (change requires restart)
[[email protected] ~]# systemctl start firewalld
[[email protected] ~]# firewall-cmd --zone=public --add-port=5432/tcp --permanent
success
[[email protected] ~]# firewall-cmd --reload
success
[[email protected] ~]# iptables -S | grep 5432
-A IN_public_allow -p tcp -m tcp --dport 5432 -m conntrack --ctstate NEW -j ACCEPT
[[email protected] ~]#

So now that we have both the engine and ovirt_engine_history in place. We need to set up the VM which will setup up the dwh service on it.

dwh service VM

$ ssh [email protected]
[[email protected] ~]# hostname
dwhservice.ovirt.org
[[email protected] ~]# yum install -y http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm > /dev/null
[[email protected] ~]# yum install -y ovirt-engine-dwh-setup > /dev/null
http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/7/x86_64/repodata/fe30bd7c1f6f8d6f4007e9096a0c0fdd305c550f8c136ba86793577edc9dc571-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/

warning: /var/cache/yum/x86_64/7/ovirt-4.1-epel/packages/libtommath-0.42.0-5.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
warning: /var/cache/yum/x86_64/7/ovirt-4.1/packages/ovirt-engine-lib-4.1.4.2-1.el7.centos.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID fe590cb7: NOKEY
Importing GPG key 0xFE590CB7:
 Userid     : "oVirt <[email protected]>"
 Fingerprint: 31a5 d783 7fad 7cb2 86cd 3469 ab8c 4f9d fe59 0cb7
 Package    : ovirt-release41-4.1.4-1.el7.centos.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-ovirt-4.1
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <[email protected]>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 From       : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
[[email protected] ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf']
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
          Version: otopi-1.6.2 (otopi-1.6.2-1.el7.centos)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment customization

          --== PRODUCT OPTIONS ==--

          Configure Data Warehouse on this host (Yes, No) [Yes]:

          --== PACKAGES ==--

[ INFO  ] Checking for product updates...
[ INFO  ] No product updates found

          --== NETWORK CONFIGURATION ==--

          Host fully qualified DNS name of this server [dwhservice.ovirt.org]:
[WARNING] Failed to resolve dwhservice.ovirt.org using DNS, it can be resolved only locally
          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]:
          The following firewall managers were detected on this system: firewalld
          Firewall manager to configure (firewalld): firewalld
[ INFO  ] firewalld will be configured as firewall manager.
          Host fully qualified DNS name of the engine server []: engine.ovirt.org
          Setup will need to do some actions on the remote engine server. Either automatically, using ssh as root to access it, or you will be prompted to manually perform each such action.
          Please choose one of the following:
          1 - Access remote engine server using ssh as root
          2 - Perform each action manually, use files to copy content around
          (1, 2) [1]:
          ssh port on remote engine server [22]:
          root password on remote engine server engine.ovirt.org:

          --== DATABASE CONFIGURATION ==--

          Where is the DWH database located? (Local, Remote) [Local]: Remote

          ATTENTION

          Manual action required.
          Please create database for ovirt-engine use. Use the following commands as an example:

          create role ovirt_engine_history with login encrypted password '<password>';
          create database ovirt_engine_history owner ovirt_engine_history
           template template0
           encoding 'UTF8' lc_collate 'en_US.UTF-8'
           lc_ctype 'en_US.UTF-8';

          Make sure that database can be accessed remotely.

          DWH database host [localhost]: dwhdb.ovirt.org
          DWH database port [5432]:
          DWH database secured connection (Yes, No) [No]:
          DWH database name [ovirt_engine_history]:
          DWH database user [ovirt_engine_history]:
          DWH database password:

          Please provide the following credentials for the Engine database.
          They should be found on the Engine server in '/etc/ovirt-engine/engine.conf.d/10-setup-database.conf'.

          Engine database host []: engine.ovirt.org
          Engine database port [5432]:
          Engine database secured connection (Yes, No) [No]:
          Engine database name [engine]:
          Engine database user [engine]:
          Engine database password:

          --== OVIRT ENGINE CONFIGURATION ==--


          --== STORAGE CONFIGURATION ==--


          --== PKI CONFIGURATION ==--


          --== APACHE CONFIGURATION ==--


          --== SYSTEM CONFIGURATION ==--


          --== MISC CONFIGURATION ==--

          Please choose Data Warehouse sampling scale:
          (1) Basic
          (2) Full
          (1, 2)[2]:

          --== END OF CONFIGURATION ==--

[ INFO  ] Stage: Setup validation

          --== CONFIGURATION PREVIEW ==--

          Firewall manager                        : firewalld
          Update Firewall                         : True
          Host FQDN                               : dwhservice.ovirt.org
          Engine database secured connection      : False
          Engine database user name               : engine
          Engine database name                    : engine
          Engine database host                    : engine.ovirt.org
          Engine database port                    : 5432
          Engine database host name validation    : False
          DWH installation                        : True
          DWH database secured connection         : False
          DWH database host                       : dwhdb.ovirt.org
          DWH database user name                  : ovirt_engine_history
          DWH database name                       : ovirt_engine_history
          DWH database port                       : 5432
          DWH database host name validation       : False

          Please confirm installation settings (OK, Cancel) [OK]:
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stopping dwh service
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up

          --== SUMMARY ==--

[ INFO  ] Starting dwh service
          Please restart the engine by running the following on engine.ovirt.org :
          # service ovirt-engine restart
          This is required for the dashboard to work.

          --== END OF SUMMARY ==--

[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170802052209-bba7e4.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170802052510-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Execution of setup completed successfully
[[email protected] ~]# service restart ovirt-engine-dwh
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.
[[email protected] ~]#
[[email protected] ~]# service ovirt-engine-dwhd restart
Redirecting to /bin/systemctl restart  ovirt-engine-dwhd.service

Now restart the service systemctl restart ovirt-engine on the testengine.ovirt.org VM and go to https://testengine.ovirt.org/ovirt-engine/

Automatinng it using Ansible

The architecture of the overall system is similar to the one above.

Setup the testengine.ovirt.org the way we did the setup in the first step.

Next would be setup the postgresql instance which would host our ovirt_engine_history, we can do

On VM C

the VM has to be setup for the ovirt_engine_history VM which is to be used by dwhservice.ovirt.org VM.

---
- hosts: dwhdb
  vars:
    # the below vars are explained in `install-postgresql/defaults/main.yml` and 
    # also the other configurable variables are placed there
    engine_vm_network_cidr: '139.162.45.0/24' # Network where the Engine VM lies
    dwhservice_vm_network_cidr: '139.162.61.0/24'
    ovirt_engine_dwh_db_password: 'password'
  roles:
    - role: ovirt-engine-remote-dwh-setup/install-postgresql

Running this would be

$ ansible-playbook site.yml -i inventory

PLAY [dwhdb] ******************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
fatal: [testdwhdb.ovirt.org]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service postgresql: host"}
...ignoring

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : install postgresql] **************************************************
changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Check if the db is initialized] **************************************
ok: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Initialize the postgresql db] ****************************************
 [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running su

changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Start postgresql.service] ********************************************
changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : creating directory for sql scripts in /tmp/ansible-sql] **************
changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : copy SQL scripts] ****************************************************
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-user-create.sql)
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-create.sql)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : create engine DWH DB and user] ***************************************
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-user-create.sql)
changed: [testdwhdb.ovirt.org] => (item=ovirt-engine-dwh-db-create.sql)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Adding engine and dwhservice vm IP's in the dwhdb conf to be acessed remotely] ***
changed: [testdwhdb.ovirt.org] => (item=host    ovirt_engine_history    ovirt_engine_history    139.162.45.0/24     md5 # engine)
changed: [testdwhdb.ovirt.org] => (item=host    ovirt_engine_history    ovirt_engine_history    139.162.61.0/24     md5 # dwhservice)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Edit the config file /var/lib/pgsql/data/postgresql.conf] ************
changed: [testdwhdb.ovirt.org] => (item=sed -i -- 's/max_connections = 100/max_connections = 150/g' /var/lib/pgsql/data/postgresql.conf)
changed: [testdwhdb.ovirt.org] => (item=sed -i "60ilisten_addresses = '*'" /var/lib/pgsql/data/postgresql.conf)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable firewalld and open up port 5432] ******************************
changed: [testdwhdb.ovirt.org] => (item=systemctl start firewalld)
changed: [testdwhdb.ovirt.org] => (item=firewall-cmd --zone=public --add-port=5432/tcp --permanent)
changed: [testdwhdb.ovirt.org] => (item=firewall-cmd --reload)

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Restart postgresql for the loading the newer configs] ****************
changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : check PostgreSQL service] ********************************************
changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : clean tmp files] *****************************************************
changed: [testdwhdb.ovirt.org]

TASK [ovirt-engine-remote-dwh-setup/install-postgresql : Enable postgresql.service to start at boot time] *********************
ok: [testdwhdb.ovirt.org]

PLAY RECAP ********************************************************************************************************************
testdwhdb.ovirt.org        : ok=16   changed=12   unreachable=0    failed=0
On VM B

Now VM which hosts the dwhservice needs to be setup.

---
- hosts: dwhservice
  vars:
    ovirt_engine_type: 'ovirt-engine'
    ovirt_engine_version: '4.1'
    ovirt_rpm_repo: 'http://resources.ovirt.org/pub/yum-repo/ovirt-release41.rpm'
    ovirt_engine_host_root_passwd: 'pycon2017'  # the root password of the host where ovirt-engine is installed
    ovirt_engine_firewall_manager: 'firewalld'
    ovirt_engine_host_fqdn: 'testengine.ovirt.org'  # FQDN of the ovirt-engine installation host, should be resolvable from the new DWH host
    ovirt_engine_db_host: 'testengine.ovirt.org' 
    ovirt_engine_dwh_db_host: 'testdwhdb.ovirt.org'
    ovirt_engine_dwh_db_password: 'password'
    ovirt_engine_history_db_on_dwhservice_host: False
  roles:
    - role: ovirt-common
    - role: ovirt-engine-install-packages
    - role: ovirt-engine-remote-dwh-setup

Running this would be

$ ansible-playbook site.yml -i inventory --skip-tags skip_yum_install_ovirt_engine,skip_yum_install_ovirt_engine_dwh -vvv

Now restart ovirt-engine in VMA by doing a systemctl restart ovirt-engine

Issues Faced

The very first thing which I faced as an issue was that, there was no proper documentation in the oVirt Downstream docs in place (as on now) in place. This made me research more about the things. Good in one way, anyway I raised report on bugzilla about here. https://bugzilla.redhat.com/show_bug.cgi?id=1475706 There was another snag like the previous one where an OTOPI variable was not being logged in the answerfiles which led me to

Reference Commits

Future Work

References

I have to say, I got stuck a lot while trying this out and there have numerous times where I was just about to give up. But my mentor was always there to support my questions and doubts. Thanks for that Lukas :)

More to come

All content is licensed under the CC-BY 4.0 License unless otherwise specified