Peppy Heppy

Tonight I attended a presentation at Google that given by Jason Hoffman (the CTO at Joyent) about Ruby, and scaling web architecture. Although none of the actual information was new to me it did remind me of the basic points to scaling a web application. Here they are.

• Scalability is language, performance, and throughput independent.

• Test each piece of your architecture and find out what the maximums of each service are.

• Find the real bottlenecks and remove them efficiently. (Use DTrace) An example was how Apache’s proxy module will limit the number of requests per second to 140 per Apache instance, and by using virtualization you could have eight instances running Apache on the same server with the capacity of 1120 rps.

Overall it was worth while especially when you factor in the snacks and dinner that Google provided.

Scale away!

In one of my last posts I showed how I was able to create completely custom urls for SEO, but there is an issue that sometimes comes up when creating custom urls or when migrating urls, etc.

Here is a simple way to ensure that urls that are being requested are valid. Google and Yahoo! (and others) crawl your sites links and can on occasion come across an incorrect ink from someone else’s site that may be old or mistyped. There are some stiff penalties associated with having two different urls pointing to the same page. There may also be a need to retire certain urls or to change the way they are formated.

Here is an example, the URL:

http://domain.com/d-123456-mountain_viewering

Should be redirected to:

http://domain.com/d-123456-mountain_view

Here is the simple solution:

I created a module that looked like the following in the lib directory and included it into the ActionController class.

include ActiveRecord

module MY
  module URL

    def page_code_object_map
      { 
        'd' => Destination, 'p' => Photo
      }
    end

    def execute_url_post_process
      canonicalize if params[:canonicalize]
    end

    def canonicalize

      whole_url   = request.request_uri().split('?')[0].split('#')[0]
      url_pieces  = current_url.split('-')
      page_type   = url_pieces[0].gsub(/\//, '')
      type_id     = url_pieces[1]

      begin
        object = page_code_object_map[page_type].find(type_id)
        canonical_url = send "custom_#{page_type}_path", object, params
      rescue RecordNotFound => e
        render :file => File.join(RAILS_ROOT, 'public', '404.html'), :status => 404
        return
      end

      if canonical_url and canonical_url != whole_url
        headers['Status'] = '301 Moved Permanently'
        redirect_to("#{http_base}#{canonical_url}", :status => 301)
      end

    end

  end
end

ActionController::Base.send :include, MY::URL
ActionView::Base.send :include, MY::URL

In the route below, notice that I am passing a parameter named :canonicalize with the value of true. This parameter is passed through to the controller as a request parameter and can be accessed in the params hash.

map.d '/d-:destination_global_id-:name*other_params', :controller => 'destinations', :action => 'show', :canonicalize => true, :destination_global_id => /\d{1,20}/, :name => /[^-]+/

How does this all work you say? Simple. In your application controller (controllers/application.rb) you need to include something like this:

before_filter :execute_url_post_process

This will start the checking process by calling the execute_url_post_process() method defined above in my module. If the route that matches passes the :cononicalize parameter, the conanicalize() method will get the current url and certain important pieces. Then depending on the object that is mapped to the page code (d) it will reconstruct the url of the destination object that should match the existing url. If it matches then were golden, if it doesn’t then we redirect to the new/correct url ensuring that we do not loose page rank or be counted as spam (duplicate content).

There are many things that you can do within this code. Some of them include managing authorization, hiding pages, etc.

I hope you enjoyed this tip. If you have any suggestions, please post them, I am sure some genius will have something to add. :)

I needed to build urls that were packed with keywords for SEO. I needed to make sure that the url more fully described the contents of the page.

This default rails url does not cut it.

/destinations/12345

This does cut it.

/d-12345-mountain_view

So here is the hack that I did to get the desired affect. (Suggestions or insults on my approach are welcomed!)

First, I added this code into a plugin that I was using for our custom routes stuff. You can probably add this to the environment.rb file or better yet to a a file within lib and just make sure that you require the file from within environment.rb. I really needed to add the ’-’ as a delimiter.

This is step is important because by default rails uses slashes (/) as a dilimeter for parts of the url, but by adding a dash (-) to the array things work the way they should.

module ActionController
  module Routing
    SEPARATORS = %w( / ; . , ? -)  
  end
end

Then I added a named route (config/routes.rb) that looked something like this:

map.d '/d-:destination_global_id-:name*other_params', :controller => 'destinations', :action => 'show', :canonicalize => true, :destination_global_id => /\d{1,20}/, :name => /[^-]+/

Now we can create helper methods that take all of these wonderful parameters.

def custom_d_path(destination, params={})
  d_path(
    destination.global_id, 
    string_for_url(destination.name)
  ) + (params.size > 0 ? create_other_parameters(params) : "")
end                                                

The method string_for_url() just replaced spaces with underscored and removed illegal characters.

The create_other_parameters() appended parameters in a subtle way that ensured that Google and Yahoo! wouldn’t get prejudice about dynamic pages with parameters. (This is another topic for another time.)

In short, now we can simply call custom_d_path(destination) from any view (or controller if we included the helper in both ActionView and ActionController classes).

I realize that there may be a better way to do this to make it simpler to code, but this is a simple example of a way to solve this problem.

Now for a couple of caveats:

1. For those who have OCRD (obsesive compulsive REST disorder) the urls may not suite your style. I use them for the read only pages of a site.
2. You may not need to go to this extreme to keyword pack your urls… there are many other approaches that may be more robust and easier to implement.

Hopefully this example helps someone. :)

On small projects I usually work right out of trunk to avoid the need to merge, but when working in teams to implement features that will be released separately creating a branch or two is the way to go. The only problem with working with branches is that you have to merge your code periodically in order to avoid nightmares. Here are the steps that I use to to a simple merge between a development branch and trunk. If there are better ways or if I missed something please let me know, but these is what worked for me.

1) First svn update local working copy (both trunk and branches)

2) Change directory to the branch (branches/development)

cd /Users/Paul/Documents/test_svn/repo/

3) Run a merge command similar to the one below as a dry-run to see if everything looks OK:

svn merge --dry-run -r 4:HEAD file:///Users/Paul/Documents/test_svn/repo/trunk

4) Then if you are satisfied with what you see, you can run the real command which will actually update your working copy with the merged files from trunk.

svn merge -r 4:HEAD file:///Users/Paul/Documents/test_svn/repo/trunk

5) If you have conflicts (lines that start with “C”,) then its time to merge the changes. I use FileMerge and merging the right version with the working version and then I save the merged file and then

6) Checkin all of the merges files by doing a svn commit.

7) No change directories to the Trunk working copy and run the following as a dry run.

svn merge --dry-run -r 4:HEAD file:///Users/Paul/Documents/test_svn/repo/branches/development

8) Then if everything checks out, you do the real merge:

svn merge -r 4:HEAD file:///Users/Paul/Documents/test_svn/repo/branches/development

NOTE: when merging the branch back into trunk, you must use the same revision number as the you did when you merged trunk into the branch, or the revision number of the commit made after the last merge from trunk to your branch.

If you not done any regular merges, which you should do BTW, to avoid really hairy merges, then your revision numbers for both merges will be the same.

9) Resolve any conflicts.

10) Checkin all of the merges files by doing a svn commit.

Now your two branches are synced up! Yeah! Happy merging!

The key is making sure you keep track of revision numbers and merging, one way to do that is to create a tag with a date or sequence number. Also, you can look into the history by using the svn log .

Here are some pretty useful tools for using FileMerge on OSX with the command line subversion.

I stumbled across these tools while I was looking around at how I can better use FileMerge with Subversion

Here is what I did to fully set them up.

# sudo su
# cd /usr/bin
# svn export http://ssel.vub.ac.be/svn-gen/bdefrain/fmscripts/fmdiff .
# svn export http://ssel.vub.ac.be/svn-gen/bdefrain/fmscripts/fmdiff3 .
# svn export http://ssel.vub.ac.be/svn-gen/bdefrain/fmscripts/fmresolve .
# exit
# vim ~/.subversion/config

Be sure to set your diff and diff3 tools to use the fmdiff and fmdiff3

Thanks Bruno De Fraine for publishing these great tools and making my life a little easier. :)

A half hour ago I released a game that I designed with the help of a friend and the input from my team members. The game took just under three weeks from inception to completion, and I also moved with my family to a new apartment as well.

It was fun to work on this project and I hope that it is successful.

Oh, on a more technical note, I wrote the game using Rails 2.0 with no plugins and only a handful of tables. It was fun on every level.

Now here comes the real question… can you beat me to where ever it is you want to go? If you are reading this, please invite me to your race (Paul Hepworth/ dderby[at]omniop[dot]com.)

Here is the link: http://realtravel.com/ddracing

You can also get to the game by going to any RealTravel destination page and click the “play” button when prompted.

Thanks again Phil for your creative flare major contribution.

Almost a year ago I was asked to work on a project with a designer to build a rails site that lists designers and their portfolios. it was a fun project overall and was during my nights and weekends over the period of a couple of months. Although my work was done for quite sometime the site didn’t technically release until a short time ago. Anyways, I coded it up and although the project was a little too large for my side-project taste it was fun.

You can check it out by going to: http://indiefashionhouse.com

Below are photos of what I went through last night while camped out and waited to register my son for an extremely popular Dual Immersion program that had only 15 open spots.

I managed to get second in line by showing up at 9pm last night. As I hung out, socialized, and slept, other parents trickled in throughout the night and morning. There were hundreds who wanted a spot and we got one, in fact we were the first to get one. Hooray! It was very refreshing to see so many parents who wanted the very best for their children and were willing to sacrifice dignity, sleep, and a warm bed—all this for their children’s education.

The temperature dropped to freezing, but I hung tight with a handful of dedicated parents. The lines actually ended up wrapping the parking lot. It wasn’t easy, but it was worth it.

Oh and I made the channel 2 news this evening. :)

A friend of mine invited me to go to the Crunchies last Friday. It was a lot of fun to be apart of the show and more than anything it was good to see winners of categories that I have never even heard of. Overall it was very inspiring. I will go next time if I get the chance.

I have way too many domain names, so that means that when I want to make a change to my zone template files including a search and replace for certain ips or just changing the email in the zone like I do below. (Or whatever you need to do.)

I first backed up my zone files with a basic but effective cp command:

blah@server ~# cp /var/named /var/named-backup

Then I replaced my email with one that would handle the spam and put it in the right mailbox (/dev/null.) :)

blah@server ~# for file in $(ls /var/named/*.db); do sed -i "s/paul.mydomain.com/dns.omniop.com/g" $file; done

Now that all of the zone files are updated, even if I were to restart my named, the files would not update my slave DNS servers because the serial number in the zones have not changed.

...
2008011502    ; serial, todays date+todays
...

So here is a quick little shell script that I wrote that increments all of my BIND zone files for my DNS server.

#!/bin/bash
for file in $(ls /var/named/*.db);
do
  if [ -f $file ];
  then
    OLD=`egrep -ho "2008[0-9]*" $file`
    NEW=$(($OLD + 1))
    sed -i "s/$OLD/$NEW/g" $file
    echo "fixed $file" 
  fi
done

There may be a better way of doing this, but I found this very quick and painless.

Now I will hopefully get less spam now that the DNS email scrapers won’t get my email from my zone files.

Hope this helps someone!

A month ago this blogs MySQL database with InnoDB tables got totally hosed due to some unfortunate events and because it was just this blog I did not have a recent sql dump backup. Anyways, thanks to Google’s index cache I was able to recover most of my blog articles. I still didn’t get all of my comments back, but hey what do you expect from a FREE blog backup service from Google. It was still better than the one that I was paying for that didn’t work. ;)

A month ago my then VPS provider, JaguarPC, has some really freaky hardware issues, that to this day I have no idea what happened and they ended up restoring a two week old backup of the whole server which included my VPS. When I fired up this blog and a couple of other sites they failed due to mysql table corruption. The corrupt databases that used Myisam tables seemed to repair just fine, but all of my InnoDB databases (Rails uses InnoDB by default when you use migrations) were unrecoverable and I ended up having to try other means for getting my data back or at least as much of my data that I could get back.

Here is what I learned:

1. Never assume that your hosts backups of your VPS will work when they are restored because they perform backups while the server is running and databases don’t like that too much.
2. Always keep backups of your databases, especially the ones that use the InnoDB table engine, in a SQL dump format.

So here is what I do now to prevent this from happening again:

1. Perform your own backups of your databases using the methods that are suggested for your db and db table engines.
2. Get the data into SQL so when your VPS is backed up it will properly backup a dump.

Assuming that you have a file that contains a list of databases with one per line, you can do something like the following and then hook up your script t cron.

#!/bin/bash

cd /var/lib/mysql

if [ ! -d sql_backup ]; then 
  mkdir sql_backup
fi

for db in $(cat databases.txt); do echo $db; mysqldump --single-transaction $db > sql_backup/${db}.sql; done

Good luck, and oh BTW, you might want to get this running on your VPS before your host does the restore. ;)

I am looking to change VPS providers from JaguarPC to a company that has a higher level of service and uptime since I have had over 4 or 5 days of downtime and even lost some data due to the restoration form backup that killed my InnoDB tables in MySQL. (enough about that)

Anyways, i found a company called InMotion and after just a few days, here is my last trouble ticket.

---

To whomever it may concern,

Please cancel my account and refund the charges to my credit card as I would like to take you up on your 30-day money back guarantee.

In case you have questions about why, here is my best effort at giving you both an explanation and some customer feedback that I hope will be helpful to you as you improve your business.

Over the past couple of days I attempted to get going with you guys in hopes that I could move my current sites/customers over to a more responsive VPS provider with better support, etc. I am sure you are a great company and provide great services to many customers, but to be honest I was not impressed with the process of signing up and how my initial tickets were handled.

1) I was surprised by the $2/ip as there is no mention of this pricing before signup and the sales page implies that you do not charge for ips as long as they are justified. Believe it or not, that was one reason why I considered you guys.

2) The support seemed very reactive overall and although I asked questions that could imply that I was relatively experienced with system administration, it seemed as though i was talked down to and was practically told to be grateful for your help and that I wasn’t charged for you services. (The “service” that I didn’t ask for of changing the port that sshd was going to run on. I only told you that I was going to be changing it in order to have you open up a hole in your firewall to accommodate my customization.)

3) By default I didn’t have root access, and after I was “approved” to have it, I was still unable to ssh into my VPS due to your network firewall. Then once I inquired about it, I had to explain that I needed ssh access and was asked for the last 4-digits of my cc number, and then the ticket was closed by a technician, as if my issue had been resolved. After opening the ticket again and providing my cc info, I was asked for the ip address that I was going to be sshing from so you could add a firewall rule that would allow me access, to which I replied that I accessed the server from many locations.

Long story short: I appreciate your help but I don’t feel like InMotion is where I should be.

Good luck and have a most pleasant day!

---

I don’t like posting negative stuff (unless its about Microsoft and Visual Studio,) but before I signed up with InMotion I searched google for stuff like “InMotion sucks”, and “InMotion bad”, etc hoping that I could see what other peoples experience were and couldn’t find much. I hope this helps other hackers who will do the same and maybe save them some grief.

And no, this was not a paid advertisement by InMotion. ;)

UPDATE: InMotion sent me an email apologizing for the miscommunications and asking me to reconsider my decision. I will admit that their email was very nice and that they wanted to keep my business and make sure that I am happy. I am sure that their services will be great for some people, personally, I am looking for a provider who is a little bit more interested in the big picture of what it is that I want to accomplish and get setup. When I have issues or problems, I need to be able to ask a specific question and get taken care of. Each support cycle is expensive for me. I think the issues that I am explaining are due to VPS companies whose primary business is shared hosting and their secondary business is VPS—their techs are trained to deal with the shared hosting folks. If I decided to stay it wouldn’t be a horrible thing I suppose, but I have already found an alternative VPS provider ServInt.

I am sure that InMotion is a good company, but it didn’t work out for me.

Last night I walked down the street with my family to view a train exhibit that had been put up year after year for over 30 years by a local train enthusiast in an effort to share his Christmas with neighbors, adults, and children alike. Two nights before that I drove by a couple of houses in Sunnyvale, CA that were covered in Christmas lights choreographed to music that could be heard by tuning your car stereo to a certain channel.

On both occasions I wondered to myself, who would have the time to spend on such projects year over year? Then I realized that this was not unlike all of the brilliant and willing software contributors who work tirelessly in open-source communities to build great software that other people can use, enjoy, and even use to make a living.

I am impressed with the passion that so many individuals have that causes them to DO what it is that they do. Its encouraging to see that people with passion can provide a great service to the rest of the world in small but meaningful ways.

While some people stand around wondering what can be accomplished or think about why things should or should not be done by them, others just DO.

So here is a thanks to all of the people that DO because they DO!

For every Ruby on Rails plug-in and gem, every Linux install and presentation given in a local user group—thanks!

So, I like many others purchased my upgrade online prior to the release date and looked forward to the 26th of October. Since I was at work I missed the package and had to go to the FedEx place after work to get it. As I was walking in a boy and his father were just leaving the cash register and the boy had a huge smile and just as they left the FedEx guy said "now don't stay up too late tonight." Then I got in line and noticed a guy behind me with a shirt with the name Geek on it and wondered if he was picking up his Leopard, but I just minded my own business and didn't ask. Long story short there were at least ten people in line now, and once I saw the size of my package and realized that everyone was picking up and walking out with the same sized package, I knew for sure that they were all there to pick up their OSX Leopard. I couldn't help but be amused that so many people were there to pick it up. The puncher was when the FedEX guy said "I can't wait until work is over and I get to go home and install mine."

Now for my quick review of my first impressions of Leopard. Overall the upgrade is pretty good, but other than the Mail upgrades with TODO's, Notes, and RSS, and maybe Time Machine, I don't think that it was imperative that I upgrade since many of the new features were already available with third-party applications like Virtue Desktops for Spaces, Time Machine and regular backup software, etc. I did like the new "Quick Look" view in Finder, but like so many of the other upgrades and improvements they are fun, cool, neat, and sweet, but not life altering. Maybe I just haven't played with it all yet and haven't looked at all of the enhancements. I don't regret the upgrade, but to all of those cash strapped people who want to upgrade because they feel like they will be left behind -- take your time.