Pkgdb2 fully setup on ireland - Official release 0.3

My goal for Release 0.3 in the beginning was to deploy Fedora Package Database and make it work with fedpkg. However, pkgdb was getting outdated so I worked on installing pkgdb2. I managed to deploy a development instance of it for my 0.1, 0.2 I could not do much as it was getting hard to communicate with Pierre.

Now, I have successfully deployed pkgdb2 on

ireland.proximity.on.ca:8080/pkgdb2

It is completely functional and nothing is broken. It has a database that it is communicating with but there is no data in it as of yet, as i did not understand all the table relationships and did not have time to look into it. Nonetheless, there is no reason why it would not work when data is put into it, because the website is connecting to the right database.

I was told to deploy the pkgdb2 on ireland on a virtual host so i set it up on port 8080. I have properly configured the firewall to allow port8080.
I requested the packages list that needed to be put into the database and I haven't got it yet, however, we have the data scheme now so that getting the packages information in there should not be a big issue. Also, if we click login on the website it is going through FAS authentication, in the future we may set up our own authentication system but for now this is working. Ideally, Anybody who is in sytem-admin or package-maintaner group in FAS should be able to modify the packages through th GUI(it is not tested yet).

The database scheme can be checked here: http://ireland.proximity.on.ca:8080/tables.txt

The database dump is also there on ireland. As of yet, the website is communicating with an empty database based on mysql.

Enjoy the holidays.

Pkgdb2- Troubleshooting the installation and Improvement suggestions

Suggestions to the Package author:

1. Please include a list of required packages in the instructions
2. Sqllite database works in development version but does not work with the deployment on Fedora19. I have checked it several times on different machines. So we can either convert the data to other databases or fix the sqllite version.
3. The package "Python Fedora" does not work properly when installed through yum. It has to be installed through github. We can include that in the instructions as well.
4. There is this line in pkgdb2.wsgi which suggests to give a path to "pkgdb2" which is actually asking the path to the clone "packagedb2". I was really confused here :). Then I got to know how python import works to figure it out.
5. Also, after installing python-flask from yum, it had to be uninstalled through pip for the FAS login to work, threebean told me that. I have no idea why.

That's about it, thank you very much for the help.

Troubleshooting installation issues:

the most annoying errors were python import errors. If you we don't know how python is working, we will be confused.

"Error importing so and so module" is just python's way of saying you have not met the dependencies and some modules are missing. Whatever module python fails to load, please check if you have installed all the required packages.

Might get some errors with mysql installations. Fedora 19 comes with Mariadb, if you are familiar with it go ahead, otherwise you will have to remove mariadb and install mysql. After installing mysql, sometimes it asks for a password and the default "mysql" password does not work. The issue has been discussed here:

http://www.webhostingtalk.com/showthread.php?t=875255

Also, when converting database, if it fails to convert. Then its probably missing the change that had to be made on the dumpfile. Regardless, if it fails, the database has to be drop and created again for the converter to work.

That's all folks :)

How to install pkgdb2 (Fedora Package Database) from scratch

This cookbook assumes that Fedora 19 is freshly installed, with some design and development tools that the Fedora setup offers.

Install the Required Packages:

$ yum install vim git python-flask \
python-flask-wtf python-wtforms \
python-kitchen python-sqlalchemy \
 postgresql-server postgresql-plpython \
python-psycopg2 mod_wsgi memcached \
 community-mysql-libs community-mysql-devel \
 community-mysqlcommunity-mysql-server \
  python-dogpile-cache python-openid-teams \
  python-openid-cla python-bugzilla python-memcached \
  python-docutils python-dateutil \
 python-urllib3 MySQL-python \
 python-pip


$ git clone https://github.com/fedora-infra/python-fedora.git

$ cd python-fedora

$ python ./setup.py install

$ cd ~

Getting the Data from the sqllite database and installing pkgdb2 libraries:

$ git clone https://github.com/fedora-infra/packagedb2.git

$ cd packagedb2

$ python ./setup.py build

if it works properly then say:

$ python ./setup.py install

you may remove the build if you want.

$ python createdb.py

$ cd /var/tmp

Converting the data dump to and inserting it into MySQL:

$ service mysqld start

$ check: tail /var/log/messages
see if it started successfully "It should say 'Started MYSQL database Server'"

$ mysqladmin -u root password yourpassword


Download this awsome sqllit to mysql script:

www.redmine.org/attachments/download/8273/sqlite3-to-mysql.py

and place it under /var/tmp

go to /var/tmp : $ cd /var/tmp

$ sqlite3 pkgdb2_dev.sqlite .dump > mydump

unfortunately, we will have to modify the dump a little bit before we can start converting:

$ vi mydump

Change this line, and two more lines

$ mysql -u root -p

mysql> create database pkgdb2;
mysql> exit

$ cat mydump | python sqlite3-to-mysql.py | mysql -u root -p pkgdb2 --default-character-set=utf8

if it executes without saying anything, that means it was successful.

Configuring pkgdb2:

Now lets get back to configuring our pkgdb2.

$ cd ~/packagedb2/

$ mv ./utility/pkgdb2.cfg.sample ./utility/pkgdb2.cfg

$ vi ./utility/pkgdb2.cfg

comment the postgres line and un-comment the mysql line. Change the admin groups to Fedora Groups you are part of, or whatever groups you want to be included in the package admin groups. Also change the FAS username and Password line to reflect your FAS credentials, it can be set to none if you do not want FAS integration. Refer to the screen-shot:

Single quotes around the values are important.




Once that is done, we will have to modify our wsgi script.
$ vi pkgdb2.wsgi

Un-comment and adjust the values as shown:

 Then we will have to edit the default-config.py script to reflect our new database:

$ vi /lib/python2.7/site-packages/pkgdb2-0.1.0-py2.7.egg/pkgdb2/default_config.py


DB_URL = 'mysql://root:mypassword@localhost/pkgdb2'

then copy the pkgdb2.conf file to http configuration directory.

$ cp pkgdb2.conf /etc/httpd/conf.d

$ vi /etc/httpd/conf.d/pkgdb2.conf

make the following changes:

Restart Apache and Mysql.
$ service httpd restart
$ service mysqld restart

Security Issues:

Open the http port:

$ iptables -I INPUT -p tcp --dport 80 -j ACCEPT

You may try to access the server, but it will give you a 403 forbidden most probably because of SELINUX or permissions missing on some of the folders. You may chose to disable SELINUX(not recommended) or make exceptions. You can make exceptions like this:

Try to access the website, get the 403 error and then come back to terminal and type

Do a tail /var/log/messages

look for this command, copy it and run on your terminal:
sealert -l a66201ea-062a-4a4b-8d2f-eaf29d3cd855

it will give you some things that are being blocked by SELINUX, and it will give you the boolean, just take the boolean and make exceptions:

setsebool -P httpd_read_user_content 1

Finishing Touch:

make sure you have execute(x) permission for everybody in the whole path to pkgdb2.cfg and pkgdb2.wsgi

also do: pip uninstall flask

and install/check if flask is installed with yum: yum install python-flask.

Now restart the services again

$ service httpd restart
$ service mysqld restart

Go to your favourite browser and type: localhost/pkgdb2
and you will see the deployement.

Any issues regarding this installation you can either ask me on the comments or directly talk to the package authors on github
they are really helpful people.

Got the database to work with pkdgb2 - \o/ -

Remember the database that I converted to mysql? yes, an innovation that I did :) I got mysql database to work with pkgdb2 so the link is not broken anymore. pkdgdb2 was not officially tested with mysql, therefore, I became the first person to do it. Although the pkgdb2 was tested with sqllite database, it was not working with it. I will update my blog with more details later on. Meanwhile, you can check the working version of pkgdb2 here: http://ireland.proximity.on.ca:8080/pkgdb2

Every Tab should work except the "login" button. That goes to FAS login, so I dont think we need to get it working.

Successfull Deployment of Fedora Package Database(pkgdb2) on ireland(could be my release 0.3 if I cannot get help to test it with a database)

As I said in my last blog post that I would try it on a new VM. I installed Fedora 19 from scratch and this time I did not install pkgdb2 under python/site-scripts, however, I ran it directly from clone to see if that makes a difference. Unfortunately, it did not make any difference. We know for sure that it is hitting the right database but somehow the connection is not opening up properly. I have left a message for Pierre on github and I am online since about 36 hours with very little sleep, waiting to catch him on IRC.

I got access to ireland on Friday, and was told by Chris to deploy the the instance on there. Therefore, however much installation was completed, I successfully set it up Fedora Package Database on ireland.proximity.on.ca. Also, I set it up on virtual-host running on port 8080. If anybody wants to check out the deployment they can do so on this url: http://ireland.proximity.on.ca:8080/pkgdb2

Keep in mind, as there is no database to test with, if you click on any link you will get a 500 internal server error as the config files are not properly communicating with the sample sqlite database. Here is a screen-shot of the instance:

 

In my desperation of testing my achievement with a database, I even converted the sample database to mysql and tried to test it with it. I thought I had to change only a couple of "dbi_url" lines, however, there was much to change in the core python scripts that I do not possess knowledge of.

If I could not test it with a database by tomorrow 6pm then I will just document the process of pkgdb2 deployment(as far as I could deploy it).

Pkgdb2 - 5 hours of head scratching till 6 in the morning

Thursday night after I finished my exams and sat down to install pkgdb2, I made some progress but got stuck in getting a database together to test the pkgdb2 with. Luckily, I found Pierre- the package author, online 1 in the morning.

I started asking him questions and we were trying to resolve why the template database that comes with the clone, not working with this installation. We checked all the important python scripts in a spree of "fpaste" then figured out that my mod_wsgi file is pointing towards the clone and not the installed pkgdb2 directory. So we fixed that and I installed Sqlite manager to see if the database is good and it was. Then we were checking the httpd logs and it still says "table Package" not found. We know from the trouble shooting "print" statements in the script that it is pointing towards the right database but still it gave the same error. Even Pierre said "Now even I don't know what is wrong" :)

Anyhow, I installed A new VM to conduct a fresh installation from scratch and see if it works this time.

Some Progress FINALLY! Fedora Package Database - should have been my release 0.2

For so long I had been stuck in getting the Fedora Package Database running. I deployed the test instance for my first release but could not do much after that as I was unable to install the Fedora Package Database. I tried to get in contact with Pierre the author of pkgdb2, he was replying on github but could not get a hold of him on IRC.

Anyhow, I sat determined today to do it myself. Started to analyze the output that I was getting when I was trying to run the setup.py scritpt. It was giving me an import error flask.ext.fas_openid module not found. I researched and found some reference to it in github on python-fedora and saw the same guys who manage pkgdb1 and 2 discussing something about that module there.

I uninstalled python-fedora and rebuilt it, but this time I got the source from github because I thought they have the most update version and I was right. I installed python-fedora from there and I stopped getting that error and my beloved pkgdb2 built successfully. Here is a screenshot of the new deployment. Notice that it is running on the IP address with the proper alias this time and not a random port number. Actually I configured the alias to be pkgdb, I can change it to pkgdb2 or anything else according to our requirements.



Now, the only thing I am missing is a database to test this deployment with. I am currently working on trying to get Pierre's script working to convert the previous packagedb database to match the new schema, I will try to use that and blog if it works.

0.2 - Pending release - waiting to communicate with Package mantainers

I did not get a chance to update my blog, however, I made some progress after 0.1. I tried to install Fedora Package database but kept running into errors that I have no clue about. It has something to do with importing the existing database to be used with pkgdb2. I am waiting to catch any of the package mantainer on IRC channel so I can install pkgdb2 while I am communicating with them. This way I don't have to report all the bugs/issues on Github or bugtrackker. This is a one time shot. I just have to catch them once and the installation can be done at that time.