Pkgdb2 fully setup on ireland - Official release 0.3

My goal for Release 0.3 in the beginning was to deploy Fedora Package Database and make it work with fedpkg. However, pkgdb was getting outdated so I worked on installing pkgdb2. I managed to deploy a development instance of it for my 0.1, 0.2 I could not do much as it was getting hard to communicate with Pierre.

Now, I have successfully deployed pkgdb2 on

ireland.proximity.on.ca:8080/pkgdb2

It is completely functional and nothing is broken. It has a database that it is communicating with but there is no data in it as of yet, as i did not understand all the table relationships and did not have time to look into it. Nonetheless, there is no reason why it would not work when data is put into it, because the website is connecting to the right database.

I was told to deploy the pkgdb2 on ireland on a virtual host so i set it up on port 8080. I have properly configured the firewall to allow port8080.
I requested the packages list that needed to be put into the database and I haven't got it yet, however, we have the data scheme now so that getting the packages information in there should not be a big issue. Also, if we click login on the website it is going through FAS authentication, in the future we may set up our own authentication system but for now this is working. Ideally, Anybody who is in sytem-admin or package-maintaner group in FAS should be able to modify the packages through th GUI(it is not tested yet).

The database scheme can be checked here: http://ireland.proximity.on.ca:8080/tables.txt

The database dump is also there on ireland. As of yet, the website is communicating with an empty database based on mysql.

Enjoy the holidays.

Pkgdb2- Troubleshooting the installation and Improvement suggestions

Suggestions to the Package author:

1. Please include a list of required packages in the instructions
2. Sqllite database works in development version but does not work with the deployment on Fedora19. I have checked it several times on different machines. So we can either convert the data to other databases or fix the sqllite version.
3. The package "Python Fedora" does not work properly when installed through yum. It has to be installed through github. We can include that in the instructions as well.
4. There is this line in pkgdb2.wsgi which suggests to give a path to "pkgdb2" which is actually asking the path to the clone "packagedb2". I was really confused here :). Then I got to know how python import works to figure it out.
5. Also, after installing python-flask from yum, it had to be uninstalled through pip for the FAS login to work, threebean told me that. I have no idea why.

That's about it, thank you very much for the help.

Troubleshooting installation issues:

the most annoying errors were python import errors. If you we don't know how python is working, we will be confused.

"Error importing so and so module" is just python's way of saying you have not met the dependencies and some modules are missing. Whatever module python fails to load, please check if you have installed all the required packages.

Might get some errors with mysql installations. Fedora 19 comes with Mariadb, if you are familiar with it go ahead, otherwise you will have to remove mariadb and install mysql. After installing mysql, sometimes it asks for a password and the default "mysql" password does not work. The issue has been discussed here:

http://www.webhostingtalk.com/showthread.php?t=875255

Also, when converting database, if it fails to convert. Then its probably missing the change that had to be made on the dumpfile. Regardless, if it fails, the database has to be drop and created again for the converter to work.

That's all folks :)

How to install pkgdb2 (Fedora Package Database) from scratch

This cookbook assumes that Fedora 19 is freshly installed, with some design and development tools that the Fedora setup offers.

Install the Required Packages:

$ yum install vim git python-flask \
python-flask-wtf python-wtforms \
python-kitchen python-sqlalchemy \
 postgresql-server postgresql-plpython \
python-psycopg2 mod_wsgi memcached \
 community-mysql-libs community-mysql-devel \
 community-mysqlcommunity-mysql-server \
  python-dogpile-cache python-openid-teams \
  python-openid-cla python-bugzilla python-memcached \
  python-docutils python-dateutil \
 python-urllib3 MySQL-python \
 python-pip


$ git clone https://github.com/fedora-infra/python-fedora.git

$ cd python-fedora

$ python ./setup.py install

$ cd ~

Getting the Data from the sqllite database and installing pkgdb2 libraries:

$ git clone https://github.com/fedora-infra/packagedb2.git

$ cd packagedb2

$ python ./setup.py build

if it works properly then say:

$ python ./setup.py install

you may remove the build if you want.

$ python createdb.py

$ cd /var/tmp

Converting the data dump to and inserting it into MySQL:

$ service mysqld start

$ check: tail /var/log/messages
see if it started successfully "It should say 'Started MYSQL database Server'"

$ mysqladmin -u root password yourpassword


Download this awsome sqllit to mysql script:

www.redmine.org/attachments/download/8273/sqlite3-to-mysql.py

and place it under /var/tmp

go to /var/tmp : $ cd /var/tmp

$ sqlite3 pkgdb2_dev.sqlite .dump > mydump

unfortunately, we will have to modify the dump a little bit before we can start converting:

$ vi mydump

Change this line, and two more lines

$ mysql -u root -p

mysql> create database pkgdb2;
mysql> exit

$ cat mydump | python sqlite3-to-mysql.py | mysql -u root -p pkgdb2 --default-character-set=utf8

if it executes without saying anything, that means it was successful.

Configuring pkgdb2:

Now lets get back to configuring our pkgdb2.

$ cd ~/packagedb2/

$ mv ./utility/pkgdb2.cfg.sample ./utility/pkgdb2.cfg

$ vi ./utility/pkgdb2.cfg

comment the postgres line and un-comment the mysql line. Change the admin groups to Fedora Groups you are part of, or whatever groups you want to be included in the package admin groups. Also change the FAS username and Password line to reflect your FAS credentials, it can be set to none if you do not want FAS integration. Refer to the screen-shot:

Single quotes around the values are important.




Once that is done, we will have to modify our wsgi script.
$ vi pkgdb2.wsgi

Un-comment and adjust the values as shown:

 Then we will have to edit the default-config.py script to reflect our new database:

$ vi /lib/python2.7/site-packages/pkgdb2-0.1.0-py2.7.egg/pkgdb2/default_config.py


DB_URL = 'mysql://root:mypassword@localhost/pkgdb2'

then copy the pkgdb2.conf file to http configuration directory.

$ cp pkgdb2.conf /etc/httpd/conf.d

$ vi /etc/httpd/conf.d/pkgdb2.conf

make the following changes:

Restart Apache and Mysql.
$ service httpd restart
$ service mysqld restart

Security Issues:

Open the http port:

$ iptables -I INPUT -p tcp --dport 80 -j ACCEPT

You may try to access the server, but it will give you a 403 forbidden most probably because of SELINUX or permissions missing on some of the folders. You may chose to disable SELINUX(not recommended) or make exceptions. You can make exceptions like this:

Try to access the website, get the 403 error and then come back to terminal and type

Do a tail /var/log/messages

look for this command, copy it and run on your terminal:
sealert -l a66201ea-062a-4a4b-8d2f-eaf29d3cd855

it will give you some things that are being blocked by SELINUX, and it will give you the boolean, just take the boolean and make exceptions:

setsebool -P httpd_read_user_content 1

Finishing Touch:

make sure you have execute(x) permission for everybody in the whole path to pkgdb2.cfg and pkgdb2.wsgi

also do: pip uninstall flask

and install/check if flask is installed with yum: yum install python-flask.

Now restart the services again

$ service httpd restart
$ service mysqld restart

Go to your favourite browser and type: localhost/pkgdb2
and you will see the deployement.

Any issues regarding this installation you can either ask me on the comments or directly talk to the package authors on github
they are really helpful people.

Got the database to work with pkdgb2 - \o/ -

Remember the database that I converted to mysql? yes, an innovation that I did :) I got mysql database to work with pkgdb2 so the link is not broken anymore. pkdgdb2 was not officially tested with mysql, therefore, I became the first person to do it. Although the pkgdb2 was tested with sqllite database, it was not working with it. I will update my blog with more details later on. Meanwhile, you can check the working version of pkgdb2 here: http://ireland.proximity.on.ca:8080/pkgdb2

Every Tab should work except the "login" button. That goes to FAS login, so I dont think we need to get it working.

Successfull Deployment of Fedora Package Database(pkgdb2) on ireland(could be my release 0.3 if I cannot get help to test it with a database)

As I said in my last blog post that I would try it on a new VM. I installed Fedora 19 from scratch and this time I did not install pkgdb2 under python/site-scripts, however, I ran it directly from clone to see if that makes a difference. Unfortunately, it did not make any difference. We know for sure that it is hitting the right database but somehow the connection is not opening up properly. I have left a message for Pierre on github and I am online since about 36 hours with very little sleep, waiting to catch him on IRC.

I got access to ireland on Friday, and was told by Chris to deploy the the instance on there. Therefore, however much installation was completed, I successfully set it up Fedora Package Database on ireland.proximity.on.ca. Also, I set it up on virtual-host running on port 8080. If anybody wants to check out the deployment they can do so on this url: http://ireland.proximity.on.ca:8080/pkgdb2

Keep in mind, as there is no database to test with, if you click on any link you will get a 500 internal server error as the config files are not properly communicating with the sample sqlite database. Here is a screen-shot of the instance:

 

In my desperation of testing my achievement with a database, I even converted the sample database to mysql and tried to test it with it. I thought I had to change only a couple of "dbi_url" lines, however, there was much to change in the core python scripts that I do not possess knowledge of.

If I could not test it with a database by tomorrow 6pm then I will just document the process of pkgdb2 deployment(as far as I could deploy it).

Pkgdb2 - 5 hours of head scratching till 6 in the morning

Thursday night after I finished my exams and sat down to install pkgdb2, I made some progress but got stuck in getting a database together to test the pkgdb2 with. Luckily, I found Pierre- the package author, online 1 in the morning.

I started asking him questions and we were trying to resolve why the template database that comes with the clone, not working with this installation. We checked all the important python scripts in a spree of "fpaste" then figured out that my mod_wsgi file is pointing towards the clone and not the installed pkgdb2 directory. So we fixed that and I installed Sqlite manager to see if the database is good and it was. Then we were checking the httpd logs and it still says "table Package" not found. We know from the trouble shooting "print" statements in the script that it is pointing towards the right database but still it gave the same error. Even Pierre said "Now even I don't know what is wrong" :)

Anyhow, I installed A new VM to conduct a fresh installation from scratch and see if it works this time.

Some Progress FINALLY! Fedora Package Database - should have been my release 0.2

For so long I had been stuck in getting the Fedora Package Database running. I deployed the test instance for my first release but could not do much after that as I was unable to install the Fedora Package Database. I tried to get in contact with Pierre the author of pkgdb2, he was replying on github but could not get a hold of him on IRC.

Anyhow, I sat determined today to do it myself. Started to analyze the output that I was getting when I was trying to run the setup.py scritpt. It was giving me an import error flask.ext.fas_openid module not found. I researched and found some reference to it in github on python-fedora and saw the same guys who manage pkgdb1 and 2 discussing something about that module there.

I uninstalled python-fedora and rebuilt it, but this time I got the source from github because I thought they have the most update version and I was right. I installed python-fedora from there and I stopped getting that error and my beloved pkgdb2 built successfully. Here is a screenshot of the new deployment. Notice that it is running on the IP address with the proper alias this time and not a random port number. Actually I configured the alias to be pkgdb, I can change it to pkgdb2 or anything else according to our requirements.



Now, the only thing I am missing is a database to test this deployment with. I am currently working on trying to get Pierre's script working to convert the previous packagedb database to match the new schema, I will try to use that and blog if it works.

0.2 - Pending release - waiting to communicate with Package mantainers

I did not get a chance to update my blog, however, I made some progress after 0.1. I tried to install Fedora Package database but kept running into errors that I have no clue about. It has something to do with importing the existing database to be used with pkgdb2. I am waiting to catch any of the package mantainer on IRC channel so I can install pkgdb2 while I am communicating with them. This way I don't have to report all the bugs/issues on Github or bugtrackker. This is a one time shot. I just have to catch them once and the installation can be done at that time.

Release 0.1.1 - Pkgdb2 succesfful deployment of temporary instance

1. Install needed packages: 

sudo yum install python-virtualenv libpq*

2. Get pkgdb2 

git clone https://github.com/fedora-infra/packagedb2.git

3. Go inside packagedb2 folder

cd packagedb2

4. run virtual env

$ virtualenv my-pkgdb2-env

5. run this command

$ source my-pkgdb2-env/bin/activate 

your prompt will change to my-pkgdb2-env now

6. Run these commands to install dependencies:

(my-pkgdb2-env)$ pip install kitchen paver urllib3
(my-pkgdb2-env)$ pip install git+https://github.com/fedora-infra/python-fedora.git
(my-pkgdb2-env)$ pip install -r requirements.txt
(my-pkgdb2-env)$ pip install -r test_requirements.txt

7. if everything went without errors then run this command to download a temporary 

database for your instance

(my-pkgdb2-env)$ ./runtests.sh 

this will take some time

8. If the result came out as "OK" then start the server:

(my-pkgdb2-env)$ python runserver.py

9. Once started, open your favourite browser and type the following URL:

http:127.0.0.0:5000

I was able to get the instance up and running. Here is a screen-shot of the new pkgdb2 

Release 0.1 - service sbrproject restart

The Main Goal

My main project was to port fedpkg to pidora so fedpkg would work with Pidora packages. For that I needed to install Fedora Package Database as fedpkg communicates with it and then modify it for Pidora packages.

Release 0.1

My goal for release 0.1 was to install Fedora Package Database(pkgdb) and make walk-through of it. I stumbled upon a few issues and was communicating with the package maintainers, they said we will only be supporting pkgdb for 2 months now and we are currently working on pkgdb2 which would be ready in 2 months. The instance of pkgdb2 that they have currently has a few bugs and is not ready to work with fedpkg.

What Now?

After a group meeting with Chris Tyler and package maintainers, we decided that I should proceed with the installation of pkgdb2, which, although is not ready yet, will be ready soon and maintained. It would not make sense to deploy a package which will go out of date very soon. Moreover, it would also not make sense to deploy pkgdb1 for Pidora as Fedora Package Database has several uses.

Fedora Package Database Installation - From Beginning to as far as I got

1.      Get the package database:

·         git clone https://github.com/fedora-infra/packagedb.git

2.      Install the needed packages:

·         yum -y install python-genshi TurboGears python-turbojson postgresql-server \

        postgresql-contrib python-TurboMail python-bugzilla python-fedora \

        python-psycopg2 python-sqlalchemy python-feedparser koji mod_wsgi \

        python-paver httpd mod_ssl babel python-cpio xz python-argparse \

        python-mako python-memcached py

NOTE: if you are running on fedora install python-fedora-turbogears by issuing command “yum install python-fedora-turbogears”

3.      go to packaged “cd packagedb”

·         python setup.py egg_info (you will get a few warnings ignore them)

           NOTE:please do a yum update before doing this.

4.     Copy paver to your current directory and renaming  it to paver.local

·         cp /usr/bin/paver paver.local

5.      Run this complicated command (copy/paste it carefully so only the code gets copied)

·         sed 's/__requires__ = .*/__requires__ = ["Paver", "CherryPy < 3.0"]/' < paver.local > paver

6.       Build the package:

·         paver build --install-conf=`pwd` --install-data=`pwd` --install-sbin=`pwd`

7.      start the postgresql service if it is not already started by issuing

·         service postgresql initdb

·         start the service using "service postgresql start"

8.      Create postgresql users: 

sudo -u postgres createuser -SDRPE pkgdbadmin

        Enter password for new role: <password>

        Enter it again: <password>

        sudo -u postgres createuser -SDRPE pkgdbreadonly

        Enter password for new role: <password>

        Enter it again: <password>

Get the most recent dump from the same website you grabbed fedora package from. so go to https://fedorahosted.org/releases/p/a/packagedb and download the most recent ".dump.xz" file

then run this command to import it all to the database:

xzcat pkgdb-20100318.dump.xz | sudo -u postgres psql

NOTE: this can take a while, no, rather this will take a while - give it about 3-4 hours if you are running it on a virtual machine.

9.       Run this command

·            sudo -u postgres psql pkgdb < update-schema/03_add_gitbranchname.sql

10.     Memcached is an optional daemon that can be run to speed up the server.  It functions by caching certain data that is read many times so that we don't have to access the database as frequently.  To use it you need to install and start it

·               su -c yum install memcached

·               su -c /etc/init.d/memcached start

11.      Move pkgdb.cfg.sample to pkgdb.cfg

·               mv pkgdb.cfg.sample pkgdb.cfg

12.      Modify the pkgdb.cfg file

·                  sqlalchemy.dburi="postgres://pkgdbadmin:(password)@localhost/pkgdb"

·            server.static_media_dir="(the path to packagedb directory e.g /home/hitman/packaged)"

13.      Modify /etc/httpd/conf/httpd.conf

·          vi /etc/httpd/conf/httpd.conf

        <Directory "/home/hitman/packagedb">

            AllowOverride None

            Options None

            Order allow,deny

            Allow from all

          </Directory>

14.    Turn SELINUX in permissive mode (just to check)

·      setenforce 0

15. Copy the httpd-pkgdb.conf file to your apache conf.d directory

·      sudo cp httpd-pkgdb.conf /etc/httpd/conf.d/

16.    Restart httpd server

·      service httpd restart

17.    ATTEMPT to access the Fedora Package Database Webpage

·         https://localhost/pkgdb/

18. Get the errors from the error log and try to resolve them.

·         tail /etc/httpd/logs/ssl_error_log

[Tue Nov 19 23:09:07.816671 2013] [:error] [pid 2494] [remote ::1:6073] mod_wsgi (pid=2494): Target WSGI script '/home/hitman/Public/packagedb/pkgdb.wsgi' cannot be loaded as Python module.
[Tue Nov 19 23:09:07.816714 2013] [:error] [pid 2494] [remote ::1:6073] mod_wsgi (pid=2494): Exception occurred processing WSGI script '/home/hitman/Public/packagedb/pkgdb.wsgi'.
[Tue Nov 19 23:09:07.816733 2013] [:error] [pid 2494] [remote ::1:6073] Traceback (most recent call last):
[Tue Nov 19 23:09:07.816757 2013] [:error] [pid 2494] [remote ::1:6073] File "/home/hitman/Public/packagedb/pkgdb.wsgi", line 24, in
[Tue Nov 19 23:09:07.816809 2013] [:error] [pid 2494] [remote ::1:6073] modulename="pkgdb.config")
[Tue Nov 19 23:09:07.816821 2013] [:error] [pid 2494] [remote ::1:6073] File "/usr/lib/python2.7/site-packages/turbogears/config.py", line 219, in update_config
[Tue Nov 19 23:09:07.816892 2013] [:error] [pid 2494] [remote ::1:6073] configdict = config_obj(configfile, modulename).dict()
[Tue Nov 19 23:09:07.816904 2013] [:error] [pid 2494] [remote ::1:6073] File "/usr/lib/python2.7/site-packages/turbogears/config.py", line 174, in config_obj
[Tue Nov 19 23:09:07.816923 2013] [:error] [pid 2494] [remote ::1:6073] modname + '.cfg')
[Tue Nov 19 23:09:07.816933 2013] [:error] [pid 2494] [remote ::1:6073] File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in resource_filename
[Tue Nov 19 23:09:07.817317 2013] [:error] [pid 2494] [remote ::1:6073] return get_provider(package_or_requirement).get_resource_filename(
[Tue Nov 19 23:09:07.817329 2013] [:error] [pid 2494] [remote ::1:6073] File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 180, in get_provider
[Tue Nov 19 23:09:07.817345 2013] [:error] [pid 2494] [remote ::1:6073] import(moduleOrReq)
[Tue Nov 19 23:09:07.817360 2013] [:error] [pid 2494] [remote ::1:6073] ImportError: No module named pkgdb

This is how far I got when installing Fedora Package Datbase.

Fedora Package Installation - Latest update November 19th

I had issue with the paver command when I was installing so I asked it on github.

https://github.com/fedora-infra/packagedb/issues/13

This was the problem:
paver build --install-conf=`pwd` --install-data=`pwd` --install-sbin=`pwd`
Traceback (most recent call last):
  File "/usr/bin/paver", line 9, in <module>
    load_entry_point('Paver==1.1.1', 'console_scripts', 'paver')()
  File "/usr/lib/python2.7/site-packages/paver/tasks.py", line 816, in main
    _launch_pavement(args)
  File "/usr/lib/python2.7/site-packages/paver/tasks.py", line 785, in _launch_pavement
    execfile(environment.pavement_file, mod.__dict__)
  File "pavement.py", line 16, in <module>
    from turbogears.finddata import find_package_data
  File "/usr/lib/python2.7/site-packages/turbogears/__init__.py", line 9, in <module>
    from turbogears.config import update_config
  File "/usr/lib/python2.7/site-packages/turbogears/config.py", line 4, in <module>
    pkg_resources.require("CherryPy<3.0")
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 690, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 592, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (CherryPy 3.2.2 (/usr/lib/python2.7/site-packages), Requirement.parse('CherryPy<3.0')

Toshio Replied today giving me a fix, so I was able to move on further. the fix was to copy paver script on the local folder and replace the version and run the paver script locally.

$ cp /usr/bin/paver paver.local
$ sed 's/__requires__ = .*/__requires__ = ["Paver", "CherryPy < 3.0"]/' < paver.local > paver
$ ./paver build --install-conf=`pwd` --install-data=`pwd` --install-sbin=`pwd`

However, now, the pkgdb-sync-yum command is broken. I have asked him again on github and waiting for his reply.

[root@localhost server-scripts]# chmod 755 pkgdb-sync-yum
[root@localhost server-scripts]# ./pkgdb-sync-yum update -h
Traceback (most recent call last):
File "./pkgdb-sync-yum", line 51, in
import pkg_resources
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2655, in
working_set.require(requires)
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: fedora-packagedb

I moved on with the steps ignoring that step and found myself having 500 error when trying to access the website. I am hoping that it is connected to the update.

RPM SIgning lab - SBR600

Followed the tutorial on the website: http://zenit.senecac.on.ca/wiki/index.php/Signing_and_Creating_a_Repository_for_RPM_Packages

executed the command gpg --key-gen,
it asked me a bunch of questions and at the end it created the RSA public key.
pub   2048R/FA60C533 2013-11-17 [expires: 2014-01-16]

after this, i just navigated to where I had my RPMS and signed it using this command

#rpm --addsign less-451-1.fc18.i686.rpm
Enter pass phrase:
Pass phrase is good.
less-451-1.fc18.i686.rpm:

here is a link to my signed RPM


[hitman@localhost i686]$ rpm --addsign grep-2.9-1.fc18.i686.rpm
Enter pass phrase:
Pass phrase is good.
grep-2.9-1.fc18.i686.rpm:
[hitman@localhost i686]$



Here is a link to my signed RPMs:


grep-2.9-1.fc18.i686.rpm
less-451-1.fc18.i686.rpm


Created the gpg key by issuing the command

gpg --export --armour msmahmood@myseneca.ca > gpgkey.txt

gpgkey.txt can be viewed here

GPG is a very handy tool, probably the only tool i know that can encrypt files, maybe there are others, however, this was pretty easy to use.

Koji Lab SBR600

I already had fedora-packager installed so I did not need to do that.



Ran the command:

/usr/bin/fedora-packager-setup 

Everything was properly set up to use koji.

GREP

[hitman@localhost SRPMS]$ time koji build f18 --scratch grep-2.9-1.fc18.src.rpm
Uploading srpm: grep-2.9-1.fc18.src.rpm
[====================================] 100% 00:00:05 1019.85 KiB 177.57 KiB/sec
Created task: 6182383
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=6182383
Watching tasks (this may be safely interrupted)...
6182383 build (f18, grep-2.9-1.fc18.src.rpm): free
6182383 build (f18, grep-2.9-1.fc18.src.rpm): free -> open (arm02-builder10.arm.fedoraproject.org)
  6182385 buildArch (grep-2.9-1.fc18.src.rpm, i686): open (buildvm-15.phx2.fedoraproject.org)
  6182384 buildArch (grep-2.9-1.fc18.src.rpm, x86_64): open (buildvm-21.phx2.fedoraproject.org)
  6182385 buildArch (grep-2.9-1.fc18.src.rpm, i686): open (buildvm-15.phx2.fedoraproject.org) -> closed
  0 free  2 open  1 done  0 failed
  6182384 buildArch (grep-2.9-1.fc18.src.rpm, x86_64): open (buildvm-21.phx2.fedoraproject.org) -> closed
  0 free  1 open  2 done  0 failed
6182383 build (f18, grep-2.9-1.fc18.src.rpm): open (arm02-builder10.arm.fedoraproject.org) -> closed
  0 free  0 open  3 done  0 failed



6182383 build (f18, grep-2.9-1.fc18.src.rpm) completed successfully



real    3m4.840s
user    0m2.116s
sys    0m0.200s

It took me about 1 minute less than then it took me to build it locally using mock. Which is a big difference considering the package is small, if it was a huge package, it probably would have taken me a bit longer.

building grep for Powerpc arch, it built it for both 32 and 64 as can be seen.

[hitman@localhost SRPMS]$ time ppc-koji build f18 --scratch grep-2.9-1.fc18.src.rpm
Uploading srpm: grep-2.9-1.fc18.src.rpm
[====================================] 100% 00:00:05 1019.85 KiB 170.42 KiB/sec
Created task: 1530737
Task info: http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1530737
Watching tasks (this may be safely interrupted)...
1530737 build (f18, grep-2.9-1.fc18.src.rpm): free
1530737 build (f18, grep-2.9-1.fc18.src.rpm): free -> open (ppc-builder2)
  1530738 buildArch (grep-2.9-1.fc18.src.rpm, ppc): open (ppc-builder6)
  1530739 buildArch (grep-2.9-1.fc18.src.rpm, ppc64): free
  1530739 buildArch (grep-2.9-1.fc18.src.rpm, ppc64): free -> open (ppc-builder2)
  1530738 buildArch (grep-2.9-1.fc18.src.rpm, ppc): open (ppc-builder6) -> closed
  0 free  2 open  1 done  0 failed
  1530739 buildArch (grep-2.9-1.fc18.src.rpm, ppc64): open (ppc-builder2) -> closed
  0 free  1 open  2 done  0 failed
1530737 build (f18, grep-2.9-1.fc18.src.rpm): open (ppc-builder2) -> closed
  0 free  0 open  3 done  0 failed

1530737 build (f18, grep-2.9-1.fc18.src.rpm) completed successfully

real    6m32.512s
user    0m5.032s
sys    0m0.338s

I did not get any errors but it did take me 6 minutes to build it.

 I am trying to build it for s390 architecture, but it gave me some error that i did not know about, now i am waiting for it to build again.

------------------------UPDATE-------------------------------------------------------------------------------
Made some changes to the spec file of grep and update the "Requires:" field

When i was building from the spec file I was noticing these "Requires:" field that I was not including but my package was still building without issues

Requires(interp): /bin/sh /bin/sh
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(post): /bin/sh info
Requires(preun): /bin/sh info
Requires: libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) libc.so.6(GLIBC_2.1.3) libc.so.6(GLIBC_2.2) libc.so.6(GLIBC_2.2.3) libc.so.6(GLIBC_2.3) libc.so.6(GLIBC_2.3.4) libc.so.6(GLIBC_2.4) libpcre.so.1 rtld(GNU_HASH)
Processing files: grep-debuginfo-2.9-1.fc18.i686
Provides: grep-debuginfo = 2.9-1.fc18 grep-debuginfo(x86-32) = 2.9-1.fc18
Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1

I searched the libraries for these packages and found out that I was not including rpm-devel and glibc. I changed my spec file and I was able to build it on s390 archs

here is a link to my new spec.file
grep.spec 
  

  
[hitman@localhost SRPMS]$ time s390-koji build f18 --scratch grep-2.9-1.fc18.src.rpm
Uploading srpm: grep-2.9-1.fc18.src.rpm
[====================================] 100% 00:00:05 1019.81 KiB 174.00 KiB/sec
Created task: 1260172
Task info: http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=1260172
Watching tasks (this may be safely interrupted)...
1260172 build (f18, grep-2.9-1.fc18.src.rpm): free
1260172 build (f18, grep-2.9-1.fc18.src.rpm): free -> open (fedora3.s390.bos.redhat.com)
  1260173 buildArch (grep-2.9-1.fc18.src.rpm, s390): free
  1260174 buildArch (grep-2.9-1.fc18.src.rpm, s390x): free
  1260173 buildArch (grep-2.9-1.fc18.src.rpm, s390): free -> open (fedora1.s390.bos.redhat.com)
  1260173 buildArch (grep-2.9-1.fc18.src.rpm, s390): open (fedora1.s390.bos.redhat.com) -> closed
  1 free  1 open  1 done  0 failed
  1260174 buildArch (grep-2.9-1.fc18.src.rpm, s390x): free -> open (fedora1.s390.bos.redhat.com)
  1260174 buildArch (grep-2.9-1.fc18.src.rpm, s390x): open (fedora1.s390.bos.redhat.com) -> closed
  0 free  1 open  2 done  0 failed
1260172 build (f18, grep-2.9-1.fc18.src.rpm): open (fedora3.s390.bos.redhat.com) -> closed
  0 free  0 open  3 done  0 failed

1260172 build (f18, grep-2.9-1.fc18.src.rpm) completed successfully

real    9m16.539s
user    0m5.653s
sys    0m0.369s 

Ok this took me about an hour to figure out and about 10 minutes to build.

LESS

Time for less!
  
 [hitman@localhost SRPMS]$ time koji build f18 --scratch less-451-1.fc18.src.rpm
Uploading srpm: less-451-1.fc18.src.rpm
[====================================] 100% 00:00:02 305.66 KiB 145.18 KiB/sec
Created task: 6182535
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=6182535
Watching tasks (this may be safely interrupted)...
6182535 build (f18, less-451-1.fc18.src.rpm): open (arm02-builder06.arm.fedoraproject.org)
  6182537 buildArch (less-451-1.fc18.src.rpm, i686): open (buildvm-21.phx2.fedoraproject.org)
  6182536 buildArch (less-451-1.fc18.src.rpm, x86_64): open (buildvm-26.phx2.fedoraproject.org)
  6182536 buildArch (less-451-1.fc18.src.rpm, x86_64): open (buildvm-26.phx2.fedoraproject.org) -> closed
  0 free  2 open  1 done  0 failed
  6182537 buildArch (less-451-1.fc18.src.rpm, i686): open (buildvm-21.phx2.fedoraproject.org) -> closed
  0 free  1 open  2 done  0 failed
6182535 build (f18, less-451-1.fc18.src.rpm): open (arm02-builder06.arm.fedoraproject.org) -> closed
  0 free  0 open  3 done  0 failed

6182535 build (f18, less-451-1.fc18.src.rpm) completed successfully

real    2m14.817s
user    0m1.682s
sys    0m0.121s

No issues yet and pretty fast as well. lets try it for other archs..

[hitman@localhost SRPMS]$ time ppc-koji build f18 --scratch less-451-1.fc18.src.rpm
Uploading srpm: less-451-1.fc18.src.rpm
[====================================] 100% 00:00:02 305.66 KiB 149.45 KiB/sec
Created task: 1530907
Task info: http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1530907
Watching tasks (this may be safely interrupted)...
1530907 build (f18, less-451-1.fc18.src.rpm): free
1530907 build (f18, less-451-1.fc18.src.rpm): free -> open (ppc-builder6)
  1530908 buildArch (less-451-1.fc18.src.rpm, ppc): open (ppc-builder6)
  1530909 buildArch (less-451-1.fc18.src.rpm, ppc64): open (ppc-builder1)
  1530908 buildArch (less-451-1.fc18.src.rpm, ppc): open (ppc-builder6) -> closed
  0 free  2 open  1 done  0 failed
  1530909 buildArch (less-451-1.fc18.src.rpm, ppc64): open (ppc-builder1) -> closed
  0 free  1 open  2 done  0 failed
1530907 build (f18, less-451-1.fc18.src.rpm): open (ppc-builder6) -> closed
  0 free  0 open  3 done  0 failed

1530907 build (f18, less-451-1.fc18.src.rpm) completed successfully

real    7m26.703s
user    0m5.605s
sys    0m0.338s

2 down, 1 to go


[hitman@localhost SRPMS]$ time s390-koji build f18 --scratch less-451-1.fc18.src.rpm
Uploading srpm: less-451-1.fc18.src.rpm
[====================================] 100% 00:00:02 305.66 KiB 140.13 KiB/sec
Created task: 1260196
Task info: http://s390.koji.fedoraproject.org/koji/taskinfo?taskID=1260196
Watching tasks (this may be safely interrupted)...
1260196 build (f18, less-451-1.fc18.src.rpm): free
1260196 build (f18, less-451-1.fc18.src.rpm): free -> open (fedora1.s390.bos.redhat.com)
  1260197 buildArch (less-451-1.fc18.src.rpm, s390): free
  1260198 buildArch (less-451-1.fc18.src.rpm, s390x): free
  1260197 buildArch (less-451-1.fc18.src.rpm, s390): free -> open (fedora3.s390.bos.redhat.com)
  1260198 buildArch (less-451-1.fc18.src.rpm, s390x): free -> open (fedora1.s390.bos.redhat.com)
  1260197 buildArch (less-451-1.fc18.src.rpm, s390): open (fedora3.s390.bos.redhat.com) -> closed
  0 free  2 open  1 done  0 failed
  1260198 buildArch (less-451-1.fc18.src.rpm, s390x): open (fedora1.s390.bos.redhat.com) -> closed
  0 free  1 open  2 done  0 failed
1260196 build (f18, less-451-1.fc18.src.rpm): open (fedora1.s390.bos.redhat.com) -> closed
  0 free  0 open  3 done  0 failed

1260196 build (f18, less-451-1.fc18.src.rpm) completed successfully

real    4m10.215s
user    0m3.400s
sys    0m0.218s

Nice, built without issues.
I am beginning to like this stuff!
I think Koji is a good tool, it makes it a lot easier to build on other architectures without actually having one, it would probably also save time for bigger packages.

Mock Lab SBR600

Mock creates chroots and builds packages in them. Its only task is to reliably populate a chroot and attempt to build a package in that chroot.

So I began by installing mock

• yum install mock

Then added my user-name to the group mock

• usermod -aG mock hitman

Then just ran the mock command to build grep under chroot.

• mock -r fedora-18-i386 grep-2.9-1.fc18.src.rpm 

 Surprisingly, It build without any issues in the first shot! Maybe because I put a lot of work in my SPEC files and made sure build requires and macros are properly given.

INFO: mock.py version 1.1.32 starting...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
INFO: Start(grep-2.9-1.fc18.src.rpm)  Config(fedora-18-i386)
Start: lock buildroot
Start: clean chroot
Finish: clean chroot
Finish: lock buildroot
Start: chroot init
Start: lock buildroot
Mock Version: 1.1.32
INFO: Mock Version: 1.1.32
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled ccache
Start: device setup
Finish: device setup
Start: yum update
Start: Outputting list of available packages
Finish: Outputting list of available packages
Finish: yum update
Start: creating cache
Finish: creating cache
Finish: lock buildroot
Finish: chroot init
INFO: Installed packages:
Start: build phase for grep-2.9-1.fc18.src.rpm
Start: device setup
Finish: device setup
Start: build setup for grep-2.9-1.fc18.src.rpm
Finish: build setup for grep-2.9-1.fc18.src.rpm
Start: rpmbuild -bb grep-2.9-1.fc18.src.rpm
Start: Outputting list of installed packages
Finish: Outputting list of installed packages
Finish: rpmbuild -bb grep-2.9-1.fc18.src.rpm
Finish: build phase for grep-2.9-1.fc18.src.rpm
INFO: Done(grep-2.9-1.fc18.src.rpm) Config(fedora-18-i386) 4 minutes 3 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-18-i386/result
Finish: run



Took me about 4 minutes 3 seconds as can be seen from the output above.

Then I ran mock to build the other package less

• mock -r fedora-18-i386 less-451-1.fc18.src.rpm

INFO: mock.py version 1.1.32 starting...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
INFO: Start(less-451-1.fc18.src.rpm)  Config(fedora-18-i386)
Start: lock buildroot
Start: clean chroot
INFO: chroot (/var/lib/mock/fedora-18-i386) unlocked and deleted
Finish: clean chroot
Finish: lock buildroot
Start: chroot init
Start: lock buildroot
Mock Version: 1.1.32
INFO: Mock Version: 1.1.32
INFO: calling preinit hooks
INFO: enabled root cache
Start: unpacking root cache
Finish: unpacking root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled ccache
Start: device setup
Finish: device setup
Start: yum update
Start: Outputting list of available packages
Finish: Outputting list of available packages
Finish: yum update
Finish: lock buildroot
Finish: chroot init
INFO: Installed packages:
Start: build phase for less-451-1.fc18.src.rpm
Start: device setup
Finish: device setup
Start: build setup for less-451-1.fc18.src.rpm
Finish: build setup for less-451-1.fc18.src.rpm
Start: rpmbuild -bb less-451-1.fc18.src.rpm
Start: Outputting list of installed packages
Finish: Outputting list of installed packages
ERROR: Exception(less-451-1.fc18.src.rpm) Config(fedora-18-i386) 0 minutes 25 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-18-i386/result
ERROR: Command failed. See logs for output.
 # ['bash', '--login', '-c', 'rpmbuild -bb --target i686 --nodeps builddir/build/SPECS/less.spec']

About time something had to go wrong. Let me check the result and modify my specs.

Did some research and found this article, they were talking about re-installing ncurses? so I figured, maybe it needs ncurses-devel. I added "BuildRequires: ncurses-devel" and ran mock again. It build without any issues.


 [hitman@localhost SRPMS]$ mock -r fedora-18-i386 less-451-1.fc18.src.rpm
INFO: mock.py version 1.1.32 starting...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
INFO: Start(less-451-1.fc18.src.rpm)  Config(fedora-18-i386)
Start: lock buildroot
Start: clean chroot
INFO: chroot (/var/lib/mock/fedora-18-i386) unlocked and deleted
Finish: clean chroot
Finish: lock buildroot
Start: chroot init
Start: lock buildroot
Mock Version: 1.1.32
INFO: Mock Version: 1.1.32
INFO: calling preinit hooks
INFO: enabled root cache
Start: unpacking root cache
Finish: unpacking root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled ccache
Start: device setup
Finish: device setup
Start: yum update
Start: Outputting list of available packages
Finish: Outputting list of available packages
Finish: yum update
Finish: lock buildroot
Finish: chroot init
INFO: Installed packages:
Start: build phase for less-451-1.fc18.src.rpm
Start: device setup
Finish: device setup
Start: build setup for less-451-1.fc18.src.rpm
Finish: build setup for less-451-1.fc18.src.rpm
Start: rpmbuild -bb less-451-1.fc18.src.rpm
Start: Outputting list of installed packages
Finish: Outputting list of installed packages
Finish: rpmbuild -bb less-451-1.fc18.src.rpm
Finish: build phase for less-451-1.fc18.src.rpm
INFO: Done(less-451-1.fc18.src.rpm) Config(fedora-18-i386) 0 minutes 30 seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-18-i386/result
Finish: run

Just took about 30 seconds. That is pretty neat!
Here is a link to my changed spec file.
less.spec

I think Mock tool is amazing, one thing I liked about it is, you can build packages on different architectures using mock which is nice.

The whole lab took be about 45 minutes to complete.

Installing Fedora Package Database

1.      Get the package database from

·         https://fedorahosted.org/releases/p/a/packagedb

2.      you will get a tarball, extract it using

·         tar -xvf fedora-packagedb-0.6.0

3.      Install the needed packages:

·         yum -y install python-genshi TurboGears python-turbojson postgresql-server \

        postgresql-contrib python-TurboMail python-bugzilla python-fedora \
        python-psycopg2 python-sqlalchemy python-feedparser koji mod_wsgi \
        python-paver httpd mod_ssl babel python-cpio xz python-argparse \
        python-mako python-memcached (you can probably copy paste this)

4.      go to fedora-packagedb-0.6.0

·         run: python setup.py egg_info (you will get a few warnings ignore them) NOTE:please do a yum update before doing this.

·         paver build --install-conf=`pwd` --install-data=`pwd` --install-sbin=`pwd`

5.      start the postgresql service if it is not already started by issuing

·         service postgresql initdb

·         start the service using "service postgresql start"

6.      Create postgresql users: 

sudo -u postgres createuser -SDRPE pkgdbadmin
        Enter password for new role: <password>
        Enter it again: <password>
          sudo -u postgres createuser -SDRPE pkgdbreadonly
        Enter password for new role: <password>
        Enter it again: <password>

Get the most recent dump from the same website you grabbed fedora package from. so go to https://fedorahosted.org/releases/p/a/packagedb and download the most recent ".dump.xz" file

then run this command to import it all to the database:

xzcat pkgdb-20100318.dump.xz | sudo -u postgres psql
NOTE: this can take a while, no, rather this will take a while - give it about 3-4 hours if you are running it on a virtual machine.

After this the instructions said to modify the HTTP server to run the python script, this is what I am having trouble with, it gives me a 403 error. and I am trying to get a hold of Raymond Chan as he is the guru when it comes to Linux System Administration. Once It is resolved I will update the blog, if anybody else have any solution please comment

#################UPDATE November 16th#################

I created an Account in Fedora and was going to ask my question, however, they suggested to ask package related questions in buzilla. I started answering questions there to the best of my ability

https://ask.fedoraproject.org/question/35812/fedora-19-etcresovconf-auto-renewed/?answer=35894#post-id-35894

I asked my question in the bugzilla but there is no package called "packagedb" there, they redirected me to ask either on git hub or email the persons responsible for the package.

https://bugzilla.redhat.com/show_bug.cgi?id=1030792


I asked on Github and I am going to send and email to a.badger@gmail.com
for the same question.

https://github.com/fedora-infra/packagedb/issues/13

#################UPDATE November 16th#################

abadger replied really quick. Check the updates on this page.
https://github.com/fedora-infra/packagedb/issues/13

###############################################################################################

paver build command gives me this. Not sure what to make of it, I will ask the package admin about this

paver build --install-conf=`pwd` --install-data=`pwd` --install-sbin=`pwd`
Traceback (most recent call last):
  File "/usr/bin/paver", line 9, in <module>
    load_entry_point('Paver==1.1.1', 'console_scripts', 'paver')()
  File "/usr/lib/python2.7/site-packages/paver/tasks.py", line 816, in main
    _launch_pavement(args)
  File "/usr/lib/python2.7/site-packages/paver/tasks.py", line 785, in _launch_pavement
    execfile(environment.pavement_file, mod.__dict__)
  File "pavement.py", line 16, in <module>
    from turbogears.finddata import find_package_data
  File "/usr/lib/python2.7/site-packages/turbogears/__init__.py", line 9, in <module>
    from turbogears.config import update_config
  File "/usr/lib/python2.7/site-packages/turbogears/config.py", line 4, in <module>
    pkg_resources.require("CherryPy<3.0")
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 690, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 592, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (CherryPy 3.2.2 (/usr/lib/python2.7/site-packages), Requirement.parse('CherryPy<3.0')