Showing posts with label logistics. Show all posts
Showing posts with label logistics. Show all posts

Tuesday, April 28, 2009

Advanced Page Numbering in Microsoft Word

For the past few weeks I've been responding to review comments on my thesis document. it boils down to a lot of tweaking of the text, and proofreading until I wish I never heard of my thesis project. But it was all pretty straight forward until tonight.

Today, in what I hope was the last review comment, I was told that the page numbering in my 262 page document was not correct. My document starts with the first page numbered 1 and proceeds to increment up to page 262.

But the thesis document requirement is:
  1. The first three pages of the document should have no page numbers
  2. starting on the 4th page pages should be numbered in roman numerals starting with the number iv. (so basically the first three pages were i, ii, and iii, but those numbers are supposed to be hidden.)
  3. Starting with the first page of Chapter 1, the pages should be numbered with incrementing Arabic numbers. 1, 2, 3...
If you, like me, are a casual Microsoft Word user you will find this task infuriating. As did the author of the Harvard Extended blog in this post. He came up with an answer, but it turns out there was a much simpler way to meet the above requirements.


The Solution
  1. Read How to control the page numbering in a Word document
  2. Find some way to thank Bill Coan for identifying this preposterous use case and writing such a useful article about how to address it.
  3. Create a new "Section" at the end of each of the first three pages using the menu Insert >> Break >> Section Break. Section breaks replace page breaks in these locations and will let you number the pages per the requirements above.
  4. Create a "Section" just before the Chapter 1 chapter heading using the same method as in step 3.
  5. Put the courser on the first page and add page numbers using the menu INSERT >> Page Numbers Be sure to un-check "Show number on the first page" just like I did the picture tot he right. Since this page is it's own section no page numbers will appear.
  6. Repeat this step for page 2 and 3 if there are page numbers showing. Those sections, may inherit the settings from page one. In that case you may not need to do anything.
  7. Put the courser on page 4 add page numbers again, but this time (per the Page Number Format image):
    1. CHECK the "Show number on first page 
    2. Choose Alignment "Center"
    3. Click Format...
    4. Choose Number Format "i, ii, iii..."
    5. Choose Start At "iv"
  8. Put the courser on the first page of Chapter 1 and add page numbers again, but this time:
    1. CHECK the "Show number on first page 
    2. Choose Alignment "Center"
    3. Click Format...
    4. Choose Number Format "1, 2, 3..."
    5. Choose Start At "1"

That's it. You should now have a document that follows the conventions for the Harvard ALM in IT Masters Thesis document.


Acknowledgments

I'd like to thank Ian Lamont, the author of Harvard Extended. I wish I had found his blog more than two weeks before I was about to finish school. And of course thanks to Google for pointing Bill Coan who wrote this obscure but brilliant article on MS Word page numbering.

Saturday, March 7, 2009

Late night thoughts and Connecting to MySql on a Remote Machine

I'm sitting here in the dark loading lots of production data into the project so I can get some realistic looking graphs for the demo of the project.

Two things occur to me at this late hour:

  1. This project while ridiculously large in scope, was a good idea. Even as I run the import script data on different sites, I'm discovering that as structured as the data is, there are still differences between the data formats and types on the legacy server. A central db and controller is long overdue.
  2. I should have written down the instructions for enabling a remote mysql client/user to connect to a mysql instance. Here they are for the next time:


Users need permission to connect to mysql on a remote linux machine. If they don't have it, trying to log in with MYSQL Admin or MYSQL Query Analyzer will fail with a 1045 error.

To fix this log into the shell on the mysql server and type this:

make sure you have granted access to your user: (run command line)
mysql> GRANT ALL ON databaseName.* TO 'your_mysql_name'@'your_client_host_or_ip';
I found these instructions @ http://forums.mysql.com/read.php?35,9919,12139#msg-12139

Wednesday, February 25, 2009

Install Instructions

I installed the project on virtual machines today.

The machines used
machine namedescription
enotify-controllerThis is the controller! rails administration installed here
enotify-gen-01generated site storage and generator code stored here and shared via nfs. generator 1 chron task runs here
enotify-gen-02mounts enotify-gen-01/generator & sites and has a chron task that runs the generator from that machine.

Once the controller and first generator are set up with nfs all future generators need nothing but to mount the generator code in gen-01 and run the generator. The controller takes care of the rest!!! So bringing new generators online means nothing more than finding a linux machine, mounting the gen-01 generator nfs mount and calling

/urs/local/enotify/generator/bin/enotify.pl --remote --gen --update

I've updated enotify.pl to use a new package called Enotify::Generator::ControllerCommunicator described here.

So the change to the legacy generator was literally the addition of about 15 lines. A new option --remote to tell the generator to call the controller, and the calls to ControllerCommunicator


The following are the install instructions for this setup. There's no reason to read this unless you're trying to reproduce it. I took notes while I set up the controller and generators. Here's exactly what I did to install them:


The Generator
Copy a legacy generator (existing code and libraries) to enotify-gen-01:/usr/local/enotify/generator such that you end up with /usr/local/enotify/generator/bin/enotify.pl

The Controller
Installing RUBY on the controller

wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6.tar.gz
tar -xzvf ruby-1.8.6.tar.gz
cd ruby-1.8.6
./configure
make
make install

ruby is installed to /usr/local/bin/ruby
* don't believe the rubyonrail.org website, there are bugs between rails 2.0.2 and ruby 1.8.7. You need 1.8.6 for this to work.



INSTALLING RUBY GEMS
wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz
tar -xzvf rubygems-1.3.1.tgz
cd rubygems-1.3.1
chmod 755 setup.rb
ruby setup.rb

INSTALLING RAILS
gem install rails -v=2.0.2

# mongrel didn't get installed, so I installed it
sudo gem install mongrel


INSTALLING THE CONTROLLER RAILS APP

#copy the controller code over to enotify-controller into /urs/local/enotify/controller

#copy the legacy site directories you want to import into
mkdir /usr/local/enotify/site_to_import


INSTALLING MYSQL
# the server
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-server-community-5.1.31-0.rhel4.i386.rpm/from/http://mysql.mirrors.pair.com/

rpm -ivh MySQL-server-community-5.1.31-0.rhel4.i386.rpm

# the client
wget http://dev.mysql.com/get/Downloads/MySQL-5.1/MySQL-client-community-5.1.31-0.rhel4.i386.rpm/from/http://mysql.mirrors.pair.com/

rpm -ivh MySQL-client-community-5.1.31-0.rhel4.i386.rpm


#In order to get a remote MYSQL Administrator to be able to connect
mysql -u root mysql
grant all privileges on *.* to 'root'@'';

# now you can connect as the root user from your pc
 

Create a new DB User match username and password to rails/controller/config/database.yml
right click on the user and set the user to be able to connect from localhost















CREATE THE RAILS DBs
Under Catalogs Choose "Create New Schema"
Create a new schema for

thesisedev_development
thesisedev_test
thesisedev_production

# Give the new user privileges
Under users click on the new user and choose the schema privileges tab.
For each thesis_dev schema add all privileges for this user.


NFS Shares and Mounts

SETUP THE NFS SHARES
On enotify-gen-01:/etc/exports add
/usr/local/enotify/generator enotify*.flubber.com(rw,no_root_squash)


SETUP THE NFS MOUNT
On the controller and every other generator add this to /etc/fstab
enotify-gen-01:/usr/local/enotify/generator /usr/local/enotify/generat
or nfs defaults 0 0


CONTROLLER DIRECTORIES
/usr/local/enotify/controller = rails app
/usr/local/enotify/site_to_import = sites that will be imported into the
controller when you run
controller/script/import/p


DATA STORE DIRECTORIES
/usr/local/enotify/generator = enotify generator code and sites.

the sites directory is [insert_your_ip_address_here]/usr/local/enotify/generator/sites because this is where the legacy generators expect to find it.


Making generated sites available from the controller

make a symbolic link that puts the sites directory under the controller public director. This lets rails serve up the static generated files. 



ln -s /usr/local/enotify/controller/public/sites /usr/local/enotify/generator/sites



DB Schema and Legacy Site Import




#Set up the schema:
rake db:migrate
# migrate in the new data
./p

# start the server
ruby script/server start


 

Saturday, February 21, 2009

Development Status Update

The blog has been quiet because I'm going flat out on development with a goal of finishing on March 1st.

The following is a quick brain dump of my progress. No promises on spelling here. It's 1:30am on a Friday after a particularly long week.


Active MQ is Gone:
I've eliminated ActiveMQ and replaced it with basic web service calls from the generators.  REST wasn't in rails the last time I developed a full app. I like what they've done with the 2.0. It has a decidedly CRUD feel all the way through.

The switch from AMQ to REST greatly simplified communications mechanism and got around a limitation in the way the rails plug-in for AMQ handles, or rather doesn't handle temporary queues.

I'll post the communication message call flow as soon as I'm done testing it (see generator perl below).

UI Update / Schema

Most of the admin UI is built. I used Active Scaffold. It took a while, but I figured out how to organize the UI work flow so that it worked around Active Scaffold limitations.

This did require a change to the schema, but it's a really good change. I was previously keeping team information in the owners table.

The Owners table had the columns project_id, person_id, manager_id. The applicaiton logic stated that if person_id == manager_id then that person was the manager of the team.

I found myself writing a lot of code to deal with this logical relationship and when I tried to make it work with active scaffold it really started to hurt. So I changed the schema.

Now there is a Team table:

class CreateTeams < ActiveRecord::Migration
  def self.up
    create_table :teams do |t|
      t.integer :project_id
      t.integer :person_id (this is the manager)

      t.timestamps
    end
  end

And the Owners Table now looks like this:
class CreateOwners < ActiveRecord::Migration
  def self.up
    create_table :owners do |t|
      t.integer :team_id
      t.integer :person_id
     
      t.timestamps
    end
  end


I deleted a tun of code and acivescaffold just works!
Convetion  over configuration every time!!!


Giving Active Scaffold a Push - Dynamic action_links

For the sake of UI consistency I did make one hack on top of Active Scaffold. I needed a link at the top of the table that had a dynamic value in it. On the teams table I wanted a link to add a new team and I needed to pass the current project ID. That requires a fair rewrite of the way active scaffold passes parameters when creating action links.

Instead of doing it the hard way I just replaced the links by modifying the resulting pages DOM with some JavaScript. http://code.google.com/p/thesisdev/source/browse/trunk/controller/app/views/teams/for_project.html.erb



CGI::Session::CookieStore::CookieOverflow
Every project has a bug that just scares the crap out of me. Getting 500 server errors because of a Cookie overflow was that bug for me. It happened at random on certain pages. It turned out to be the way I was using Active Scaffold. 

I was rendering with this line 
:active_scaffold => 'teams', 
:constraints => {:project_id => @project}
Apparently Active Scaffold stores constrains in the session object. So after refreshing a few times and the leaving and returing to this page BOOOOOMM!!! All that project info was stored in the cookie and it overflowed its 4k limit.

The fix was simple. Just change @project to @project.id so the new code is


:active_scaffold => 'teams', 
:constraints => {:project_id => @project.id}
And that get's rid of the cookie overflows.


Active Scaffold Sortable is a lie!

There's this amazing screencast showing a rails plugin that allows you to drag and drop records to reorder them. When I install the plugin I can't even get the server to start. I've tried every permutation of rails/active_scaffold version I can think of, posted emails on forums and even emailed the developer.  Result = silence.

Based on the other forum posts it appears the code worked for a brief window when stars aligned and no one touched the active_scaffold code base. That time passed, and no one tagged the source tree, so it is lost forever.  Sadly I'm giving up on sortable, and will have to figure out some other way to order bug lists and table column headers.
 

Generator Perl:
The perl package that enables the generator to talk to the controller via REST is written.  Testing with the controller commences tomorrow.

I still have to integrate this with the generator, but I had a look through the generator logic and it's all neatly contained in a single well documented script. [Thank you me from 5 years ago!]  Once the controller is tested I'll be able to drop the communication package in, and add/edit a few simple lines, to make the controller the master of the generators workload.

Deployment:
Three Red Hat Linux Virtual Machines should be available early next week so I can test the system in a truly distributed fashion.


What's Left in Development:
  1. Error checking logic for adding teams, owners and administrators
  2. Changing the generator to listen to the controller
  3. Controller / generator integration
  4. Controller UI for generators and generations
  5. Cloning Projects, Bug_Lists and Teams
  6. Lots of testing
  7. Deployment to the test bed

Once it's deployed I'm going to start writing my paper. The applicaion will have 3 weeks of real life soak time while I'm writing.

While I don't plan to deploy it in production before I finish the thesis, it should be very obvious if there are any problems with the system. 

Monday, February 9, 2009

Active MQ and the mysterious XTools

Tonight was a night of minor frustrations. Working on my thesis was postponed by my daughter who decided bed time should be moved to 9:30pm. She's very cute, but I can't work on my thesis when she's around because she insists on my reading her books. If I try to develop while reading to her I end of with variable names like Bert and Ernie.

Once she went to sleep:

I installed ActiveMQ version 5.2
  1. download
  2. untar
  3. cd [AMQ install director]/bin
  4. chmod 755 activemq
  5. cd ..
  6. bin/activemq
And it just works.... It even has a web admin running at http://localhost:8161/admin/

Then I tried to get a simple perl script to connect via Net::Stomp.

I tried to install Net::Stomp from using sudo -H cpan -i Net::Stomp and it failed because make was missing from my machine.

For the next hour my brain didn't function as I search google for a way to install make on OS X. The search terms "install make on OS X" are not helpful...

Eventually I found and was simultaneously directed to Apples XTools. Duh... I should have looked on apple's site first.

I've now downloaded the svelte 1GB XTools installer, but I'm to tired to actually run it. That's a task for tomorrow.

Saturday, February 7, 2009

22 Days of Development Left

My thesis is due April 1st. That means the running code and thesis paper which is likely to run about 75 pages have to be done.

I'm giving myself till March 1st to finish the code...

And then it's on to the paper...

Textmate +Mac = no time to write

Using Textmate, I find myself with no down time when I develop. With Apenta and Ruby on Windows I frequently found myself with entire minutes where my IDE or scripts were unresponsive... During those times I made updates to this blog.

With my new environment MAC + TextMate, there is no down time. As a result I have a big backlog of updates. Topics include:
  • Changes to the communication workflow
  • Rendering generator understandable XML from the controller
  • Unit Testing
  • Separating test and production migration fixtures
  • Setting up the table relationships in the models
  • Deployment environment progress update
Details coming soon. Right now TextMate is calling...

Sunday, February 1, 2009

Switching to a Mac


I've been developing with RoR -v=2.02 on my winxp laptop and I'm running a scripts very frequently.

rake db:migrate:reset and ruby script/runner script/import/project.rb and most importantly rake test:units.

Running these commands is slow.

Painfully slow.

So slow so that I took out a watch and timed it.

It takes 29 seconds for the environment to spin up and then between 1 and 3 seconds to run the actual scripts. You can tell because the first thing most of these scripts do after requiring the framework is printing to standard out. But nothing prints for 29 seconds!!!

So that is 29 seconds of nothing and then a flash of my program running. It got really tedious so I decided to investigate. I started with 2 premises:
  1. The average developer would probably not put up with this and still say Rails was cool
  2. I don't remember Rails being like this when I developed on linux.
Google showed me a pile of web pages where people were complaining about rails scripts being slow. They all had one thing in common... They were running windows.

So I decided to burn an hour and try out my project on my wife's Mackbook Pro. And the scripts scream. That 29 second delay is eliminated. And the added bonus is, TextMate is very slick and does not spike the CPU the way Aptana Studio does when I say, click on a tab.

So I'm switching to my wife's mac. I'm assuming she'll like this because it means I'll finish my thesis about 3 years faster.

Wednesday, January 14, 2009

Too many rows for history!

In the legacy system " Histories" are used to store the bug count for the cross section of each owner, buglist and project each time the site is generated. This information is used in historical reporting to show how a person's, team's, or project's bug count for a given bug list has changed over time.
I sat down to write the DB import code for bug count histories tonight and noticed I didn't have a table for it. Suspicious...
So I looked at my old notes and discovered a calcuation I did a while back to figure out the number of rows I'd need in my DB.
The Math
Assuming 100 active sites/projects

(active project_people ~ 30,000) * (active bug lists ~ 1000) * (1 year of active data ~365 days) == 10,950,000,000 (Time to call Carl Sagan)

I did a few quick searches and found answers suggesting there was no theoretical limit to DB size as long as you have a nudlear powered CPU, infinate memory and a heat sink liquid cooled by the nothern Atlantic ocean.
BUT... They all talk about things in terms of millions of rows and keeping individual tables under 500 million rows to keep from encouraging Loki (god of fire and trickery) from laying waste to their application.
Additional review of my calculations shows:



Descriptionrow count% of DB
Sum of all rows except bug counts:428,0580.003909%
Sum of History Rows for 1 year:10,950,000,00099.9960909%

The purpose of the Central Controller was to have a central point for site management. Not to act as a historical data repository that will someday rival amazon.com's.

OK that's an exaggeration, but it occurs to me that if I store histories in the DB I'm leaving a ticking time bomb for whoever takes over the app. First it the central controller will start to slow down and about a 1.5 years after we deploy it will either crash the server or heat it up so hot that it will melt a hole through the outer mantle of the planet.
Either way, I'm not doing anyone any favors writing this import code.
The Solution

Given:
  • Histories have only one purpose, graphing historical bug counts.
  • I'm planning on replacing the graphing tools using a javascript based graphing tool
  • That graphing tool is going to want historical data in Javascript
I'm going to:
Not import histories into the DB. Instead I'll store the historical data in JSON as flat files on the web server with the other generated site content.
To do that I need to:
  1. Figure out the best Object Notation for the graphing tool I'm going to use.
  2. Write a one time script that writes converts the existing historical data into JSON files
  3. Teach the generator how to read and re-render those files each time it runs.
So I'm moving the histories import mechanism into the next phase when I work on the generator.

Wednesday, January 7, 2009

The Aptana Saga Continues

The developers over at Aptana are on the ball. They've been looking into the bugs I logged ROR-1097 and ROR-1098

So far they haven't found the thing that's causing the CPU to spike on startup and when I mistakenly click on the Rake Tasks tab. But I'm getting by. When the bug hits I just take a coffee break.

I was having trouble being specific about the bug behavior since it manifests for 6 minutes and most of the time the behavior is spiked CPU and nothing else. So today while I was in a meeting and my PC was idol, I fired up Camtasia and made a screen recording of the bug.

Warning! This is really boring to watch.

Saturday, October 11, 2008

Two Problems

One of the things I knew and still ignored was checking in untested code is a bad idea. Compound that by ignoring said code for a few months allowing myself to forget the context of what I had done and I have a recipe for putting this project back together like Humpty Dumpty.

Technical Background:


To do the data import I run the command script/runner scripts/import/project.rb. That's not the problem, but it's important information. project.rb does the following

thesis_importer = ThesisImporter.new("directory for the legacy data files")
thesis_importer.load_project_data
ThesisImporter is a file in app/helper that holds the logic for how to parse the existing data files, putting the data in Model Objects, and saving them to the DB.

I have two problems:


1) Logging
The code in ThesisImporter used to be in a plain old script where I created a logger that I used throughout the code
logger = Logger.new(STDOUT)
 In theory when I run the script using script/runner there should be a logger available in the environment. BUT,  as soon as the process gets to the first line with a logger...
logger.info("this thing doesn't work at all")
...I get an error saying logger is an undefined variable. I know it's been a while but I thought the rails environment set up logger for me. So why can't I use it in a helper I'm calling from a script in with script/runner?



2) Database
So far I've been using the rails embedded sqllite DB, but it was an enormous pain to see any of the data in that database, so I decided to switch to mySQL. I installed MYSQL5.0, changed the encoding of both rails and mysql to "utf8" and fired it up. I was able to create the schema in mysql by calling

script/runner db/schema.rb
Now, if if go back to running my project.rb script above (commenting out ll the logging and replacing it with "puts" then I get an error as soon as I try to save an object where Bitnami Ruby Stack complains that I'm not using a production mysql binding in rails and I need to install the C binding for mysql. My code stops working there, so I try installing the C binding.

gem install mysql

The doc insstal appears to fail, but everything else (the actual binding install) works... I restart everything... Then when I run the project.rb script again I get the error saying I'm still using the wrong binding.

OK. It just occurred to me I'm dealing with two MYSQL instances. One from Bitnami and the one I installed last night. It's possible I'm doing the gem install that is somehow specific to the bitnami MYsql instance. However, my rails app is pointing at the NEW mysql 5.0 install from last night. I'll try pointing at the bitami install tonight.



Conclusion
I should never leave broken code alone for more than a few hours. Context is more useful than all the tea in china.

I still have no idea why the logging error is happening. I need to go look through some old rails code to see if I did something special to make logger.info available.

The MySql problem could be an issue with having two mysql installations that are fighting with each other.

I can't wait to get past these environment issues and get back to the actual project. More as this develops.

Wednesday, July 23, 2008

4th of July Update

Development Environment



Over the 4th of July weekend I had a bunch of time to work on my thesis under some strange conditions.



  • In a house in upstate NY


  • Babysitting my sleeping kids

  • No internet connection

  • Forgot all my reference books

  • Listening to the Spiderwick chronicles on my ipod







Data Import



My past experience has shown me that there is no substitute for real data in development and testing, so I'm focusing on getting the data imported from the legacy system into the new rails based central controller.



The data is realitively well structured in XML and csv files, but it is organized differently than it will be in my new relational DB which means there some work to do transforming the data as I import it.



So Far I've completed the import code for:

  1. Basic project/site meta-data 

  2. Bug list definition

While listening to the Spiderwick Chronicles with no access to reference material I stubbed out the code for importing:

  1. Owner & Team Lists

  2. Bug list Projections

There's a lot of work to do building the relationships between the tables and I've captured that in  FIXME: comments in the code.



Source Code



Having deleted useful code more than once in the short time I've been working on this project I decided it was time to start using source control.  Once I got back to an internet connection I set up a repository.



I'm hosting the project on google code at http://code.google.com/p/thesisdev/ and using the subclipse plugin to handle checkins.



The one thing that bugs me is I checked in non-working code. It's not something I like doing, but I really didn't want to lose any of the Spiderwick work. 



My next job is to clean up all the pseudo code from the 4th and finish the import scripts.

Friday, June 27, 2008

First Line of Code

In the last two weeks I successfully:
  1. Got my development environment set up
  2. Created the model and migrations written based on the schema I've designed.
  3. Prototyped the migration script
Dev environment: I’m using Aptana Studio ( the evolved form of RadRails for Eclipse ) for an IDE. It’s come a long way since RadRails. There are a lot of IDE choices out there. I chose Aptana because most of the developers I know are using some form of Eclipse, so this gives me a chance to try out all the cool plugins they're always talking about.  I also installed the Bitnami Ruby Stack which includes MySQL.

Creat Models & Migrations: I created a rails project in Aptana and constructed the scaffold (down to migrations) for each of the tables in my schema.



Some of the scaffolds will be thrown away (join tables) but it’s useful to be able to view and manipulate data from the UI for the moment.

I made a few simple changes/notes to the schema.

  1. In the projects table start_date and projection_start_date seem to be duplicates. I need to go back and validate the meaning of each of those rows
  2. I added an application_id column to data tables such as the bug_tool_urls table. This is a table that has data populated once and it rarely (almost never) changes. Essentially it contains constants. The application_id column is meant to give me strict control over the ID that other products will use to join to this table. I’ve run into a problem where the default id column remembers the last rows ID even if I delete that row. So this makes it difficult for me to easily map to the URLs when I’m importing data.
  3. Even as I added the column I thought of a few ways to easily eliminate the need for the application id. I have a hunch it won’t be long lived.

So the migrations are written and I created a SQLite3 DB using them. I’ll switch to MYSQL when I get closer to a fully functioning system.

Data Import: I also prototyped scripts for migrating the existing text and xml based site data into the DB. I’ve created a helper class that has all of the brains of the migration. It contains POC code that:

  1. Shows how to traverse the directory structure of an existing site
  2. Read XML site properties using REXML
  3. Put those properties into a new Project model and save them to the DB.

The trick to making this work is to run the script using script/runner which spins up the rails environment for this project and gives me access to the DB through the Project model using ActiveRecord. It's a fully script-able environment with all the ease of rails based DB access!


Next steps

#1 Complete Data Import 
Everything else depends on the existence of data in the DB. If I proceed with another part of the project first I’ll need to build mock data. Given that I have loads of production data, it’s probably faster and less error prone to finish the data import and use real data when developing the other parts.


#2 Prototype the communication channel between the Central Controller and Generators.

This has three parts

  1. Integrating ActiveMQ with the existing Generator.
  2. Integrating ActiveMQ with the Rails Application
  3. Prototype messaging between the systems

Once the systems are talking I’ll real data sets to make sure message load and sizes are reasonable. I’ve already thought through this, so what I really need is the real data so I can run some realistic tests.

The Rails Administration portal GUI – While this will be fun, the scaffolds are sufficient for supporting the development of the other portions.

The data import and AMQ work will take me through to the begging of August at which time I’ll assess next steps.

Thursday, June 12, 2008

Back to Work After Moving

I'm back on the job after moving to a new town. My thesis adviser gave me 4 actions a while back. Here's the result:

1) Send Schema
http://docs.google.com/Doc?docid=dfn4hjr3_156gsj5k89b&hl=en

I posted this once before, but have since made one major update.




I'm leaning towards generating graphing data dynamically from the DB. This means I need to store the bug counts for the cross section of each bug list for each person at least once a day. See the the Table: project_person_bug_list_count section of the schema document for details.


2) Total quantity of data - # of rows
http://spreadsheets.google.com/ccc?key=pwxRLPxLbuoIXbKN4R_3Meg

The spreadsheet shows each table and the expected number of rows it will contain. I was very conservative with my estimates and I'd be stunned if the app grew this much. That said, if I was good at estimating application demand, I wouldn't be doing this project in the first place.

The key take away from the information is that all of the configuration and site rendering records for 500 projects comes to ~ half a million rows. BUT, when I add in the record of the bug counts for every person and every list (see schema doc) that adds 11 billion rows over the course of a year.

Is 11 billion rows a lot in terms of a MySQL table?

If so I'll look into a way to collapse that data. With some post processing it would be possible reduce the number of rows to one per person per bug list by creating a comma seperated string of bug counts.


3) Block or word diagram on how to aggregate data
http://docs.google.com/Doc?id=dfn4hjr3_174gq9pm9gq

This is a pretty basic explanation of what happens when you try to collect bug data.


4) Create a realistic schedule - research and deliverables
I've created a google calendar. There's not much on it yet.
My goal is (in the next two week) to:
  1. Get my development environment set up
  2. Create the model and migrations written based on the schema I've designed.
  3. prototype the migration script
One I have the migration script I'll have a real life test of mapping the existing data from the legacy system to the new system. This should uncover any major gaps I have in transitioning between the systems.

Following that I'll go after one of the following:
  1. Prototype messaging layer for communication between the central controller and the generator
  2. Research changes required to the generator
  3. Research access to NFS
  4. Development of Central Controller site configuration pages

Thursday, May 1, 2008

Officially Starting

My thesis proposal has been accepted and I'm registering for my last class, which is basically my thesis class.

That gives me a maximum of 9 months or until February 12th, 2009 to finish my thesis and get a grade.

Hello World

Looks like everything is working.