Wednesday, February 25, 2009

Success!

Tonight it was time to take the controller and generators for a spin and see if this thing was going to work in real life. Once the install was complete:

I fired up the controller...
ruby /usr/local/enotify/controller/script/server start

And kicked off a generator on a separate machine..
cd /usr/local/enotify/generator

perl ./bin/enotify.pl

The generator called out to the Controller and asked for a site to generate, the Controller responded with a site. The generator generated the site, checking in and updating the controller on it's progress.

When it finished the site was available through the controller web server!

http://enotify-controller/sites/faq/output/website/index.html

SUCCESS!!! 
The hard part of the development is DONE!!! The dev tasks that remain are:
  • adding the controller scheduler logic (basically translating this from the legacy system)
  • adding the replacement graphing code (all JS based and prototyped)
  • Admin UI CSS for cleaning up the look and feel
  • ldap integration for creating owner lists.
  • testing

Install Instructions

I installed the project on virtual machines today.

The machines used
machine namedescription
enotify-controllerThis is the controller! rails administration installed here
enotify-gen-01generated site storage and generator code stored here and shared via nfs. generator 1 chron task runs here
enotify-gen-02mounts enotify-gen-01/generator & sites and has a chron task that runs the generator from that machine.

Once the controller and first generator are set up with nfs all future generators need nothing but to mount the generator code in gen-01 and run the generator. The controller takes care of the rest!!! So bringing new generators online means nothing more than finding a linux machine, mounting the gen-01 generator nfs mount and calling

/urs/local/enotify/generator/bin/enotify.pl --remote --gen --update

I've updated enotify.pl to use a new package called Enotify::Generator::ControllerCommunicator described here.

So the change to the legacy generator was literally the addition of about 15 lines. A new option --remote to tell the generator to call the controller, and the calls to ControllerCommunicator


The following are the install instructions for this setup. There's no reason to read this unless you're trying to reproduce it. I took notes while I set up the controller and generators. Here's exactly what I did to install them:


The Generator
Copy a legacy generator (existing code and libraries) to enotify-gen-01:/usr/local/enotify/generator such that you end up with /usr/local/enotify/generator/bin/enotify.pl

The Controller
Installing RUBY on the controller

wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6.tar.gz
tar -xzvf ruby-1.8.6.tar.gz
cd ruby-1.8.6
./configure
make
make install

ruby is installed to /usr/local/bin/ruby
* don't believe the rubyonrail.org website, there are bugs between rails 2.0.2 and ruby 1.8.7. You need 1.8.6 for this to work.



INSTALLING RUBY GEMS
wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz
tar -xzvf rubygems-1.3.1.tgz
cd rubygems-1.3.1
chmod 755 setup.rb
ruby setup.rb

INSTALLING RAILS
gem install rails -v=2.0.2

# mongrel didn't get installed, so I installed it
sudo gem install mongrel


INSTALLING THE CONTROLLER RAILS APP

#copy the controller code over to enotify-controller into /urs/local/enotify/controller

#copy the legacy site directories you want to import into
mkdir /usr/local/enotify/site_to_import


INSTALLING MYSQL
# the server
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-server-community-5.1.31-0.rhel4.i386.rpm/from/http://mysql.mirrors.pair.com/

rpm -ivh MySQL-server-community-5.1.31-0.rhel4.i386.rpm

# the client
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-client-community-5.1.31-0.rhel4.i386.rpm/from/http://mysql.mirrors.pair.com/

rpm -ivh MySQL-client-community-5.1.31-0.rhel4.i386.rpm


#In order to get a remote MYSQL Administrator to be able to connect
mysql -u root mysql
grant all privileges on *.* to 'root'@'';

# now you can connect as the root user from your pc
 

Create a new DB User match username and password to rails/controller/config/database.yml
right click on the user and set the user to be able to connect from localhost















CREATE THE RAILS DBs
Under Catalogs Choose "Create New Schema"
Create a new schema for

thesisedev_development
thesisedev_test
thesisedev_production

# Give the new user privileges
Under users click on the new user and choose the schema privileges tab.
For each thesis_dev schema add all privileges for this user.


NFS Shares and Mounts

SETUP THE NFS SHARES
On enotify-gen-01:/etc/exports add
/usr/local/enotify/generator enotify*.flubber.com(rw,no_root_squash)


SETUP THE NFS MOUNT
On the controller and every other generator add this to /etc/fstab
enotify-gen-01:/usr/local/enotify/generator /usr/local/enotify/generat
or nfs defaults 0 0


CONTROLLER DIRECTORIES
/usr/local/enotify/controller = rails app
/usr/local/enotify/site_to_import = sites that will be imported into the
controller when you run
controller/script/import/p


DATA STORE DIRECTORIES
/usr/local/enotify/generator = enotify generator code and sites.

the sites directory is [insert_your_ip_address_here]/usr/local/enotify/generator/sites because this is where the legacy generators expect to find it.


Making generated sites available from the controller

make a symbolic link that puts the sites directory under the controller public director. This lets rails serve up the static generated files. 



ln -s /usr/local/enotify/controller/public/sites /usr/local/enotify/generator/sites



DB Schema and Legacy Site Import




#Set up the schema:
rake db:migrate
# migrate in the new data
./p

# start the server
ruby script/server start


 

Tuesday, February 24, 2009

Generator Controller Communication

In my last post I mentioned that I took ActiveMQ out of the project and settled on web services for communicating between the Generators an the controller, but I left out how the communication would work.


I settled on a very simple model. Generators are responsible for asking for work and attempting to update the controller with generation status. The Controller is responsible for doling out work, tracking progress, error management and and dealing with rogue generators.

Rogue generators aren't likely given their stability over the last 5 years, but if they were to go bad they would do things like:
  • Ask for a site to generate. Not complete generating the site.  Ask for another site to generate.
  • Fail to report a status update for generating a site.
In these cases, the controller has the error checking capabilities to clean up the generation data as much as possible and either tell the generator to do nothing, re-render a site, or render a new site.

The Perl Doc for the module used by the generator to communicate with the controller is probably the best way to understand how this works.

=head1 Enotify::Generator::ControllerCommunicator

  This package enables communication with the enotify controller
  web application.
 
  The functionality in this package supports the communication model
  where the generator ALWAYS initiates contact with the Controller.
  The process flow is as follows
 
  1) a cron task kicks off the generator
  2) the generator asks the controller what work to do
  3) the controller responds with the name of a project
  4) the generator generates the site & sends status messages
     to the controller during the update process
  5) the generator finishes the site update, notifies the controller and exits
     

the cron tab is on a 2 minute retry, with logic to keep multiple generators from starting. so step 1 will happen at most 2 minutes after step 5 completes
 

  What does the controller do with the status updates.
   
    The controller uses these status updates to record the progress of the generator in the Generation object. Each status message corresponds to a column in the generation table. The Controller records the current time in that columns when it recieves a status
    message.
   
    Status updates are simply messages saying the generator is starting or ending some part of the site generation. For example:
       
    GENERATION_BUG_LIST_DOWNLOAD_START means the generator is about to start downloading
    the sites bug lists and GENERATION_BUG_LIST_DOWNLOAD_END means the generator finished
    downloading those bug lists.

    Complete list of status messages
 
        GENERATION_START
        GENERATION_END
       
        GENERATION_BUG_LIST_DOWNLOAD_START
        GENERATION_BUG_LIST_DOWNLOAD_END

        GENERATION_RENDERING_PAGES_START
        GENERATION_RENDERING_PAGES_END

        GENERATION_RENDERING_GRAPHS_START
        GENERATION_RENDERING_GRAPHS_END

   The controller identifies all requests based on the IP address of the
   generator which is automatically passed with every call.
  
   Deployment Requirement:
     Because it is technally possible for a DHCP generator to change it's IP
     address while running a site, the enterprise deployment of this mechanism
     requires generators with Static IP addresses.
    
    
   Example use:
  
    $controller_comunicator = Enotify::Generator::ControllerCommunicator::new();
    
    $site_name = $controller_comunicator->get_site_to_generate()
   
    print "Site Name " . $site_name;
   
    >Site Name foo
   
    $controller_comunicator->
    
=cut

=head2 ControllerCommunicator->get_site_to_generate() {{{

    Purpose: This method requests a new site to generate from the
              controller.
            
    Expects: Nothing. Seriously, it is the controllers responcibility
             to deal with generators that go crazy or ask for this
             info multiple times.
                 
    Returns: (STRING) The short name of the site that is being updated. This
             is also the last directory in the path to the
             directory for the site.
            
             print "enotify/sites/" . $controller_comunicator->get_site_to_generate()
   
             > enotif/sites/foo
=cut
=head2 ControllerCommunicator->update_generation_progress() {{{

    Purpose: This method sends a status update to the controller.
           
                Valid Status messages:

                        GENERATION_START
                        GENERATION_END

                        GENERATION_BUG_LIST_DOWNLOAD_START
                        GENERATION_BUG_LIST_DOWNLOAD_END

                        GENERATION_RENDERING_PAGES_START
                        GENERATION_RENDERING_PAGES_END

                        GENERATION_RENDERING_GRAPHS_START
                        GENERATION_RENDERING_GRAPHS_END
         
            
    Expects: The name of the update to send.
                 
    Returns: (BOOLEAN) True if the update succeeded. False if there was a server error.
            
             print  $controller_comunicator->get_site_to_generate(GENERATION_START)
   
             > 1
=cut

Saturday, February 21, 2009

Development Status Update

The blog has been quiet because I'm going flat out on development with a goal of finishing on March 1st.

The following is a quick brain dump of my progress. No promises on spelling here. It's 1:30am on a Friday after a particularly long week.


Active MQ is Gone:
I've eliminated ActiveMQ and replaced it with basic web service calls from the generators.  REST wasn't in rails the last time I developed a full app. I like what they've done with the 2.0. It has a decidedly CRUD feel all the way through.

The switch from AMQ to REST greatly simplified communications mechanism and got around a limitation in the way the rails plug-in for AMQ handles, or rather doesn't handle temporary queues.

I'll post the communication message call flow as soon as I'm done testing it (see generator perl below).

UI Update / Schema

Most of the admin UI is built. I used Active Scaffold. It took a while, but I figured out how to organize the UI work flow so that it worked around Active Scaffold limitations.

This did require a change to the schema, but it's a really good change. I was previously keeping team information in the owners table.

The Owners table had the columns project_id, person_id, manager_id. The applicaiton logic stated that if person_id == manager_id then that person was the manager of the team.

I found myself writing a lot of code to deal with this logical relationship and when I tried to make it work with active scaffold it really started to hurt. So I changed the schema.

Now there is a Team table:

class CreateTeams < ActiveRecord::Migration
  def self.up
    create_table :teams do |t|
      t.integer :project_id
      t.integer :person_id (this is the manager)

      t.timestamps
    end
  end

And the Owners Table now looks like this:
class CreateOwners < ActiveRecord::Migration
  def self.up
    create_table :owners do |t|
      t.integer :team_id
      t.integer :person_id
     
      t.timestamps
    end
  end


I deleted a tun of code and acivescaffold just works!
Convetion  over configuration every time!!!


Giving Active Scaffold a Push - Dynamic action_links

For the sake of UI consistency I did make one hack on top of Active Scaffold. I needed a link at the top of the table that had a dynamic value in it. On the teams table I wanted a link to add a new team and I needed to pass the current project ID. That requires a fair rewrite of the way active scaffold passes parameters when creating action links.

Instead of doing it the hard way I just replaced the links by modifying the resulting pages DOM with some JavaScript. http://code.google.com/p/thesisdev/source/browse/trunk/controller/app/views/teams/for_project.html.erb



CGI::Session::CookieStore::CookieOverflow
Every project has a bug that just scares the crap out of me. Getting 500 server errors because of a Cookie overflow was that bug for me. It happened at random on certain pages. It turned out to be the way I was using Active Scaffold. 

I was rendering with this line 
:active_scaffold => 'teams', 
:constraints => {:project_id => @project}
Apparently Active Scaffold stores constrains in the session object. So after refreshing a few times and the leaving and returing to this page BOOOOOMM!!! All that project info was stored in the cookie and it overflowed its 4k limit.

The fix was simple. Just change @project to @project.id so the new code is


:active_scaffold => 'teams', 
:constraints => {:project_id => @project.id}
And that get's rid of the cookie overflows.


Active Scaffold Sortable is a lie!

There's this amazing screencast showing a rails plugin that allows you to drag and drop records to reorder them. When I install the plugin I can't even get the server to start. I've tried every permutation of rails/active_scaffold version I can think of, posted emails on forums and even emailed the developer.  Result = silence.

Based on the other forum posts it appears the code worked for a brief window when stars aligned and no one touched the active_scaffold code base. That time passed, and no one tagged the source tree, so it is lost forever.  Sadly I'm giving up on sortable, and will have to figure out some other way to order bug lists and table column headers.
 

Generator Perl:
The perl package that enables the generator to talk to the controller via REST is written.  Testing with the controller commences tomorrow.

I still have to integrate this with the generator, but I had a look through the generator logic and it's all neatly contained in a single well documented script. [Thank you me from 5 years ago!]  Once the controller is tested I'll be able to drop the communication package in, and add/edit a few simple lines, to make the controller the master of the generators workload.

Deployment:
Three Red Hat Linux Virtual Machines should be available early next week so I can test the system in a truly distributed fashion.


What's Left in Development:
  1. Error checking logic for adding teams, owners and administrators
  2. Changing the generator to listen to the controller
  3. Controller / generator integration
  4. Controller UI for generators and generations
  5. Cloning Projects, Bug_Lists and Teams
  6. Lots of testing
  7. Deployment to the test bed

Once it's deployed I'm going to start writing my paper. The applicaion will have 3 weeks of real life soak time while I'm writing.

While I don't plan to deploy it in production before I finish the thesis, it should be very obvious if there are any problems with the system. 

Monday, February 9, 2009

Active MQ and the mysterious XTools

Tonight was a night of minor frustrations. Working on my thesis was postponed by my daughter who decided bed time should be moved to 9:30pm. She's very cute, but I can't work on my thesis when she's around because she insists on my reading her books. If I try to develop while reading to her I end of with variable names like Bert and Ernie.

Once she went to sleep:

I installed ActiveMQ version 5.2
  1. download
  2. untar
  3. cd [AMQ install director]/bin
  4. chmod 755 activemq
  5. cd ..
  6. bin/activemq
And it just works.... It even has a web admin running at http://localhost:8161/admin/

Then I tried to get a simple perl script to connect via Net::Stomp.

I tried to install Net::Stomp from using sudo -H cpan -i Net::Stomp and it failed because make was missing from my machine.

For the next hour my brain didn't function as I search google for a way to install make on OS X. The search terms "install make on OS X" are not helpful...

Eventually I found and was simultaneously directed to Apples XTools. Duh... I should have looked on apple's site first.

I've now downloaded the svelte 1GB XTools installer, but I'm to tired to actually run it. That's a task for tomorrow.

Saturday, February 7, 2009

22 Days of Development Left

My thesis is due April 1st. That means the running code and thesis paper which is likely to run about 75 pages have to be done.

I'm giving myself till March 1st to finish the code...

And then it's on to the paper...

Lots of progress

Since the last time I wrote about actual feature development a lot has happened.

application_id is no more
Per my previous post on application_ids I decided to get rid of them. They are now gone. That means updates to
  • migrations (removed all the application_id
  • model relationships (has_many and belongs_to are not straight forward)
  • the legacy importer now looks up IDs and assigns them instead of using the application_id
Fixtures
I'd been using test/fixtures via rake db:fixtures:load to import some baseline data and constants into the dev and production DB. Once I started writing tests this didn't make sense anymore because the test fixtures were clobbering the production DB.

So I created a db/fixtures directory and told the migrations to load those fixtures. Now the baseline data is imported into the DB when I run rake db:migrate:reset

Communication with the Generator

As I thought though the communication from the controller to the generator I realized that I was going to have to rewrite the generator property loaders in order to read whatever format I wrote the site properties in from the controller.

Dreading that task I had a thought. Why not just have the controller render the site properties and other generator config data in the exact format the generator was expecting. So I did.

Taking it one step further I realized there was no reason to send this info over ActiveMQ. Since the controller and all the generators are connected to the same file share, I simply told the controller where the site directory was and then had it write the property files needed by the generator.

Arguably, this is going full circle. I just did all this work to import all that information out of the legacy files and into the DB, and now I'm taking it out of the DB and putting it back into the files!

But the benefit is a rapid development and test environment. It's trivial to bring the existing generator up and running since the site directory looks EXACTLY the way the generator expects. This allows me to focus on the main goal of the project. Centralizing control of the application and generation scheduling in the controller.

At this point the controller knows how to:
  1. Create a site directory from scratch
  2. publish site_properties.xml
  3. publish queries.xml
  4. publish the owner lists
The upshot of this dev work is it forced me to build all the relationships between most of the Models in order to be able to generate the content of those files.

Skipping Projections
I did NOT publish projections yet. Graph generation is one of the few parts of the generate I'm changing. I'll likely dynamically generate projections from the controller so they are always up to date. I'm leaving projections until I tackel the graphing part.

thesis.next
It's time to get ActiveMQ up and running for communications between the controller and generators.

Textmate +Mac = no time to write

Using Textmate, I find myself with no down time when I develop. With Apenta and Ruby on Windows I frequently found myself with entire minutes where my IDE or scripts were unresponsive... During those times I made updates to this blog.

With my new environment MAC + TextMate, there is no down time. As a result I have a big backlog of updates. Topics include:
  • Changes to the communication workflow
  • Rendering generator understandable XML from the controller
  • Unit Testing
  • Separating test and production migration fixtures
  • Setting up the table relationships in the models
  • Deployment environment progress update
Details coming soon. Right now TextMate is calling...

Sunday, February 1, 2009

Switching to a Mac


I've been developing with RoR -v=2.02 on my winxp laptop and I'm running a scripts very frequently.

rake db:migrate:reset and ruby script/runner script/import/project.rb and most importantly rake test:units.

Running these commands is slow.

Painfully slow.

So slow so that I took out a watch and timed it.

It takes 29 seconds for the environment to spin up and then between 1 and 3 seconds to run the actual scripts. You can tell because the first thing most of these scripts do after requiring the framework is printing to standard out. But nothing prints for 29 seconds!!!

So that is 29 seconds of nothing and then a flash of my program running. It got really tedious so I decided to investigate. I started with 2 premises:
  1. The average developer would probably not put up with this and still say Rails was cool
  2. I don't remember Rails being like this when I developed on linux.
Google showed me a pile of web pages where people were complaining about rails scripts being slow. They all had one thing in common... They were running windows.

So I decided to burn an hour and try out my project on my wife's Mackbook Pro. And the scripts scream. That 29 second delay is eliminated. And the added bonus is, TextMate is very slick and does not spike the CPU the way Aptana Studio does when I say, click on a tab.

So I'm switching to my wife's mac. I'm assuming she'll like this because it means I'll finish my thesis about 3 years faster.