blog.petersen.vg

Life Scientists and Social Media

Thu, 27 Mar 2008 00:00:00 -0700

I just got an email from BioInformatics LLC advertising their new new report, The New Collaboration: Social Media and the Life Science Opportunity. It is a great report, and is closely related to what we are doing here at the Assay Depot. A few things jumped out at me while reading it.

Product Information

According to the report, third party online portals are the least trusted source of product information, while company websites are the most trusted. I can understand that, when I’m looking for a computer, I check out Apple’s website before I check Amazon. Now I wouldn’t characterize Amazon (or the Assay Depot for that matter) as online portals, but we are both marketplaces that act as third parties in transactions between customers and companies.

To mitigate this trust issue, and to make things as easy as possible for our users, wherever possible, we will link the products on our website to the companies own web page about that product.

Objective Feedback

If social media isn’t the most trusted source of product information, it is the most trusted source for objective feedback. 45% of respondents said so. This complements our strategy of providing peer reviews of the services and products we offer.

Influence on Purchasing Decision

Section 8 of the report touches on how social media has influenced purchasing decisions. Interestingly it seems that social media has made them more informed regarding their purchasing decisions, but it hasn’t made purchasing faster, or changed the purchasing process.

I strongly believe this is going to change. Currently their are no tools that combine social media and purchasing, it is a bit of a disjointed process. People search for information (often using social media) about what they are purchasing. Then they go back to the old system of calling to negotiate prices and legal contracts and faxing purchase orders back and forth. I’m trying not to make this post too much of a “sales pitch”, but that is what we are trying to do. Make the purchasing of research services easy by combining e-commerce with customer reviews, links to the actual supplier, and eventually more community and social media capabilities. We just want to make scientists lives easier.

Finally…

There are two more quotes in the report that I found interesting:

Social Media appeals to the fundamental values of science - communication contribution and collaborating.

I couldn’t agree more…

and:

In an environment where the average scientist buys products from a dozen or more vendors, making it easier for them to open a dialog with you can can only strengthen their loyalty.

right on again…

Create Your Own EC2 Image

Mon, 10 Mar 2008 00:00:00 -0700

We have been working day and night preparing for our beta launch in April, which is just one of the many excuses I have for not posting in so long. Part of preparing for that launch is setting up the platform that we are going to deploy to. For numerous reasons, we’ve chosen Amazon’s EC2 for our production system. EC2 is a great service, and getting up and running with it isn’t hard at all. However, when it comes to creating your own image, I ran into some barriers, I thought I’d share how we overcame them. I hope you find it useful.

(Side note: Deepak Singh over at business|byte|genes|molecules blogged about an interview with Jeff Barr, an Amazon Web Services Evangelist which I found interesting).

Assumptions

For the purpose of this post, I am assuming your have a working EC2 account, your rsa key is installed properly, and have the basic tools working, ie: Here is a good tutorial on how to get up and running.

Starting with EC2 on Rails

In my search for a deployment platform, I came across the EC2 on Rails project. It has a bunch of nice things built in, but for various reasons, their standard image wouldn’t work for us. Nevertheless, we decided to use their image as our starting point. You can get their current AMI’s by running the following command (assuming you have the ec2onrails gem installed): Currently, the 32-bit AMI is ami-e620c58f. Next we start an instance and login: In order to rebundle the image, you will need your pk and cert on the server. I store mine in the /mnt/aws-config directory, so lets create that directory. From the server run: Next, upload the files from your local computer To make things easier in the coming steps, I add the pk and cert filenames to my environment. On the server, edit your .profile to include the following lines (I add them before ./usr/local/ec2onrails/config, but I’m not sure it matters): Next we have to prepare to rebundle. The EC2 on Rails image comes with a rebundle script. However, I was never able to make it work, so I broke out the necessary steps here:

Insert your configuration here

At this point, you are ready to configure the sever however you see fit. For us it was installing some support software, etc. Although you may want to complete the tutorial first and make sure you can rebundle properly before investing too much time!

The /mnt directory

So at this point you have your perfect server, all you need to do is store the image so you can preserve it forever. There is one last item to deal with, the mnt directory. Whenever you ec2kill an instance and start a new one from the image, the /mnt directory get cleared, which makes sense. You need some place to store all the transient data (like logfiles and deployment versions) that don’t belong in the image. However, somethings in there (like directories for log files) need to exists for the image to start properly. So what’s a person to do? Well I decided on a scheme where I tar up a skeleton on the mnt directory, and wrote an install script for it. That way when I start a new instance, I log in, run ./install.sh and I’m up and running.

Create your mnt directory skeleton

Next I copy the /mnt directory to my home directory and get it just the way I like it (at least get rid of the openssh_id.pub and lost+found). Now tar up your directory: Next we need to write the install script. Mine looks something like (don’t forget to make it executable):

That’s it, your instance is now configured and ready, now for the good stuff.

Finally, its time to bundle

First we create the image: Next, we need to upload the image to S3, (you must create an S3 bucket prior to this step): The final step is to register your image with Amazon and get your AMI:

Putting it all together (AKA, the test)

So you’ve now created your bundle. The true test is, can you start it, login, run your install script, and have a fully functional server. From your local machine: Now run the install script from the remote machine: You should now have a fully functional server.

Bonus

I created two aliases on my local machine that i found very helpful, maybe you will too:

Coming Attractions

Like I said earlier, we are launching our beta in April, so expect to see a lot of changes to the site shortly. One of those changes will include moving from WordPress to Mephisto. if your feed reader resets, I apologize in advance.

Geocoding in Ruby

Wed, 21 Nov 2007 00:00:00 -0800

The Geocode Module

Have you ever needed to turn a partial address into a full one? Have you ever wanted to get the latitude or longitude for a particular address? Maybe you want to plot addresses on a Google Map, or maybe, like a colleague of mine, you find it redundant to ask for city, state AND zip when creating a form. If any of these apply to you, you might be interested in geocoding.

Geocoding is the process of turning an address (or in some cases a partial address) into latitude and longitude. Often times during the process you get more information about the address. For instance if you geocode “1000 5th Ave, 92101”, not only with you get the latitude and longitude (32.715714, -117.160158), you also find out that it’s in San Diego, CA.

There are many services out there that will geocode addresses for you. For instance the Cartographer plugin provides a couple of nice Geocoding modules. One for Geocoder.us and one for Ontok. Both are great alternatives for geocoding. However, I wanted to use Google, and I didn’t want to use the full Cartographer plugin. So I wrote the following module that allows you to geocode addresses in Ruby using Google’s service.

You’ll notice a few returned items that are always nil, that’s because it is a drop in replacement for the Geocoding modules in Cartographer. So even if you use the Cartographer plugin, you can still use this module.

Using the Geocode Module

To use the Geocode module, simply include it, and pass it an address or partial address.

Stats on Rails

Sun, 18 Nov 2007 00:00:00 -0800

The Stats Module

Have you ever had an array of numbers and needed to know if one was a statistical outlier? Ever needed the standard deviation of those numbers? Well, I was recently working on an app, where I did. I packaged up the functionality as the following Module.

If you are unfamiliar with the grubbs test… well you’re not alone, neither was I. I found this page to be especially helpful.

Usage

To use the module, simply include it, calculate mean, stddev and n, then use the outlier? funtion.

Instantly add wiki functionality to your Rails app

Sun, 07 Oct 2007 00:00:00 -0700

Introduction

Its been a while since we’ve posted. That’s because we’ve been busy preparing for our beta launch. Toward that end, we’ve hired Rob Kaufman of Notch8 to strengthen our development effort. This post is about one of the plugins that has come out of that work, wiki_column.

wiki_column, allows you to add wiki functionality to any (textual) attribute of any model. Wiki links in that text will then refer to the model that you define as your wiki, the wiki model defaults to WikiPage. Any attribute you define as a wiki_column, will also get textile formatting automatically, for this reason wiki_column requires the RedCloth gem.

How to install it

Let’s assume you want to add a wiki to your Rails app (why else would you be here?). Let’s also assume that you have a Product model that has a description attribute that would like wiki enabled. Given those assumptions, we’ll start at the beginning. First install the RedCloth gem:

Next install the wiki_column plugin.

Next generate your WikiPage model and your Product model.

Don’t forget to migrate.

Now we add the wiki functionality to your models, and validate that our slugs are unique. In your WikiPage model, add the following lines.

Similarly, add the following line to your Product model.

Next we need to change the show views. wiki_column adds a new method to your model called wiki_#{column}. So in your show.rhtml for WikiPage, change:

Do the same for Product descriptions so they will link directly to your new wiki.

Lastly, want our url to make sense to users. Our scaffold resource is called WikiPage, because we are adding/editing and viewing individual pages. However, to a user viewing our site, the following url would make more sense.

Thus, the last step is to add a new resource to the routes.rb file

That’s it, you should now have a functioning wiki and your products should integrate with it nicely.

How to use it

Now that your application is wiki enabled, how do you make use of this new found functionality?

You already know you can use the full range of textile commands, they do a better job of explaining their syntax than I could.

What wiki_column adds is the ability to specify wiki words. Those wiki words should be surrounded by square braces. For example, if you wanted to link to a page with the slug ProductionInformation, your code would look like

Conclusion

We designed wiki_column to be easily integrated into new or existing Ruby on Rails applications, as well as to be easy for your customers to use. As alway your comments, critiques, suggestions, and especially patches are welcome. Please post below.

ActiveCalendar Update, Validators and added Flexibility

Sat, 11 Aug 2007 00:00:00 -0700

Introduction

We posted the ActiveCalendar project a couple of months ago (incidentally it was the first post on this blog as well). Since then, the response has been great. It seems like people are actually using it. For that, thank you, we’re very happy people have found this code useful. Some of you have been kind enough to post comments and questions. In response, we’ve been making incremental changes (hopefully improvements) to the code base. Rather than continue to detail them in the comments of the original post, I decided roll them up and describe them here.

validates_as_date

The plugin originally left the text fields uneditable, so the only way to edit the date was through the calendar. It became clear that this wasn’t ideal. Not only is it sometimes quicker to type your date out, as Art pointed out, once you selected a date, there was no way to blank it out. To alleviate this problem, we simply made the text fields editable. However, this create another potential problem, invalid dates. This is where validates_as_date was born.

To validate your dates, add the following code to your model:

That’s it! validates_as_date is a slight adaptation of the code snippet by Stuart Rackham found here.

Add Dynarch Parameters

The Dynarch Calendar is very flexible… up until recently our plugin was not. Many of you wanted to access the flexibility, but were unable to. We recently checked in some code that allows you to pass options to the date_select and datetime_select methods and have the options passed into Dynarch’s Calendar.

For instance, Mike Figley wanted to use the date status function, which allows you to specify some dates as special. Now you can, so using the example from Dynarch’s documentation, you could now use ActiveCalendar to highlight certain dates <%= f.date_select :arrived_on, :dateStatusFunc => "dateStatusHandler" %> ]]>

Notice the last line, now any extra options are passed directly to the Calendar.setup method.

Included in the list of possible options you can pass is ifFormat, which is the date format ActiveCalendar uses. Uma Shankar Ladha wanted to be able to just select a month and year, I’m not sure if ActiveCalendar is the best way to do that, but you could use the following code to make sure that no matter what date the user selects, your program would only get the month and year:

That format is used by both Ruby and Javascript for displaying dates, so you have to make sure that your format is parsable by both Ruby and the Calendar Javascript. Luckily their formats are extremely similar. The following list contains all the tags recognizable by both:

In case you are interested the Ruby formats you can’t use are:

And the Dynarch codes you can’t use are:

Also note that if you change the formatting, validates_as_date will not work properly. I plan on fixing that in the future.

Overridable IDs

The last change worth mentioning (there have been others) was contributed by Peer Allen. He was using ActiveCalendar in conjunction with AJAX and needed to override the id property. His code allows the programmer to specify the name and id of the html entries used by ActiveCalendar. Thanks Peer!

Conclusion

Thank you all for using and contributing to ActiveCalendar. Please let us know how you’re using it and what problems you’ve encountered.

IE PNG Fix in Rails

Thu, 02 Aug 2007 00:00:00 -0700

Introduction

PNGs are great, having a full 8 bits of alpha transparency makes designing web pages so much easier. That being said, I’m sure anyone reading this post is well aware of the gross inadequacies of Internet Explorer 6 in terms to the rendering of PNGs. IE7 has remedied the situation, however, according to Google Analytics, 6% of the visitors to this website still use IE6, and the situation is much worse for our company website where a full 37% of visitors still use IE6. My point is, if we want to take advantage of the features PNGs give us, we have to account for the IE6 users.

We are in luck, koivi.com has a write up on how we can use the Microsoft specific “DXImageTransform.Microsoft.AlphaImageLoader” css filter to load PNGs properly. The short version is, our image tags that used to look like:

must now look like (for IE6 only):

To make this process easier, I’ve written a Ruby on Rails method that handles the details for us.

png_tag

png_tag is the little brother of image_tag. It is actually two functions, first a function (ie?) to determine if the client is using IE, and if so what version. The second function (png_tag) acts just like image_tag, except when passed a png it will automatically use the DXImageTransform.Microsoft.AlphaImageLoader filter if the client is using IE6.

The first function is as follows:

This is a pretty rudimentary check, it just looks for “MSIE” in the user agent, if it finds it it returns the number after it (the version) otherwise it returns nil. Now that we have the ability to see if our user is using IE, we can tackle the second function:

If this code looks a lot like the code for “image_tag” that because it basically is “image_tag” with the IE6 check. If we find IE6, we simply change out the image source for a spacer and add the filter code. Since it also checks that the source image is a PNG before adding the filter, you can use png_tag everywhere you would have used image_tag.

My Image Doesn’t Show Up in IE?!?

I forgot to mention, there is one caveat. The “DXImageTransform.Microsoft.AlphaImageLoader” doesn’t automatically size your “img” tag. So your image is showing up, it just happens to be 1 pixel by 1 pixel (the size of our spacer). This just means you have to specify to the size of your image ahead of time (a good practice anyway).

In the case where the programmer doesn’t specify the size, you could use this code to determine it at run time. I decided not to for my code, but keep that in mind if it is a feature you really need.

Using png_tag

I thought about bundling this as a plugin, but it just seemed like overkill. If you want to use png_tag, simply copy the two functions above into your application_helper.rb file, and copy the following spacer into your images directory:

right click here to save the 1x1 pixel spacer image.

I hope you find this code useful!

Setup a Central Logging Server using Splunk, syslog-ng and named pipes on Ubuntu

Sun, 29 Jul 2007 00:00:00 -0700

Introduction

As you may already know, that I love Virtual Computing. My program of choice is VMWare, but regardless of what you use, I just love the concept. Virtual computing allows a small company like the Assay Depot to create many virtual computers, each handling a different aspect of the business. However, nothing is free in this world, and Virtual Computing comes with its own set of challenges. In this post, I hope to show how you can use Splunk, sysnlog-ng and named pipes to keep you informed as to what is happening on each of those virtual computers (it works just as well for physical computers). I use Ubuntu, so this post assumes you do too. You may have to do some interpreting if this is not the case.

During this project I found this post informative: http://mysfitt.net/tutorials/splunk_fifo.php

Design

I have many Ubuntu machines, database servers, development servers, mail relays, application servers, etc. I want to monitor them using Splunk. Splunk is a program that reads log files, makes them searchable and generally more manageable. Splunk can read log files from many different sources, for this post, I chose to to tail files from the local file system.

Since I want to manage all my machines with one instance of Splunk, my next challenge was to continuously feed log files from those machines onto one central logging server. This is where syslog-ng comes in. On one end, syslog-ng can be configured to send all system log statements to a network port. On the other end, it can be configured to receive those statements and write them to a file. In this way, our central logging server can handle all the system log files of all the Ubuntu machines on our network and Splunk can index them all.

Finally, I want to monitor services that don’t write directly to the system log. I would like to have syslog-ng simply tail those log files that I’m interested, however, either syslog-ng can’t do that, or I couldn’t figure it out (probably the latter). Instead it wrote a Ruby daemon that will tail a file and pipe the results to a named pipe. Syslog-ng then reads from that named pipe and sends the information over a network port to the central logging server that writes the statement to a file for indexing.

Simple enough, lets get to it.

Install Splunk

For this post, I’ve chosen the Splunk 3.0 beta. If you choose to use version 2.2.6, you can download the deb file. For the beta, I downloaded the tgz version.

The following code should be executed on the machine you plan to use as your central logging server.

That’s it, we downloaded the package, expanded it to /opt, copied the init script to /etc/init.d and made it executable, told the server to start it when the machine starts, and finally started the server. Since we used the defaults (installed /opt/splunk) we didn’t have to change anything. If you want to install to a different location, their README is very informative.

Install syslog-ng (Server)

Next we need to syslog-ng on the central logging server and configure it to accept logging statements from the clients.

First, install syslog-ng using apt-get:

by default, syslog-ng doesn’t use dns names for performance reasons, but we want to turn it on so our logs are stored in directories named by server name, rather than by IP address. To turn it on, edit /etc/syslog-ng/syslog-ng.conf and change the line that says: to

Next, you have to set up some sources and destinations. Sources are where syslog-ng will read from and are typically named with a leading “s_”. Destinations are where syslog-ng will write to and are typically named with a leading “d_”. Log statements connect sources and destinations. I added the following to my /etc/syslog-ng/syslog-ng.conf file.

The source reads logging statements from port 5140. The destination writes those logging statements to files in /u01/log/hosts based on the host name, date and which service published the logging statement. Lastly, the log statement connects the two.

Once you restart the syslog-ng service, your central logging server is ready to go:

Install syslog-ng (Client)

Now that you have completed your central logging server, its time to setup some clients. First (just like with the server) you need to install syslog-ng:

Now setup some sources and destinations in /etc/syslog-ng/syslog-ng.conf, except this time, you will be reading log statements from the system and writing them to a network port:

The destination statement sets up syslog-ng to write to port 5140 on the server splunk01 (that’s the name of my central logging server). The log statement connects s_all, which is the default source for syslog-ng, to your logging server.

Your logging server which should now be collecting all kernel log messages from all the servers you’ve configured.

Log Custom Services

Ok, so far we are logging all kernel log statements to your logging server. That’s great, but we really want to log our custom services which write to standard log files. Services such as postgres or Rails. Now, I’m certainly not a syslog-ng expert, but I couldn’t figure out how to get syslog-ng to tail a file, I could get it to read a file once, but that didn’t help much. I did however get syslog-ng to read from a named pipe. So I took it upon myself to write a program that would tail a file to a named pipe. (BTW, a named pipe, aka a FIFO, is tool for interprocess communication, one process can write to it like a file, while another reads from it like a file).

First, we create our named pipe:

I called mine, /var/syslog-ng/syslog_fifo. Now lets make syslog-ng read from that fifo and write to our network port. First we add a new source to /etc/syslog-ng/syslog-ng.conf

Next we update the log statement from the previous section, it should now look like:

Now, it is reading from the s_all source as well as our new s_pipe source and writing everything to d_splunk, our logging server.

You must restart syslog-ng for the changes to take effect:

Finally, we need to tail the files we are interested in and pipe them to our new fifo. To accomplish this I wrote the following daemon using Ruby:

The daemon reads from /etc/rlogd.conf. The config file consists of the filenames of any file that you want to be tailed separated by newlines. For instance, if you wanted to monitor your postgres server, your rlogd.conf file would look something like:

Now start the rlogd server:

At this point, your client is setup. It should be sending all kernel log messages, as well as messages from log files you’ve configured to your logging server where they are stored and indexed.

Since I will be performing this on all of my servers, I went ahead and packaged it up as a deb file, so setting up my clients is as easy as:

Configure Splunk

Your Splunk server should already be running and your clients should be logging to it. The last step is to configure Splunk to read the log files. First, visit http://splunk01:8000/admin, where splunk01 is the name of your logging server. Next, click on “Data Inputs”, then on “Files and Directories”. From there you can “Add Input”; Set the path to /u01/log (or whatever directory you are using) and change the host to “Regex on path”. I use the following regex:

Build Your Own Deb File and Repository Server

Mon, 09 Jul 2007 00:00:00 -0700

Introduction

How do you install software on an Ubuntu machine? How do you create a deb file? How does apt-get work? These are the questions I will try to answer in this post, but first a little background.

Background

I love Virtual Computing. I can run tons of virtual servers on a very limited number of physical machines. This allows me to segregate applications exactly as I choose. I can have subversion and Trac running on one and nothing but an email gateway on another, it also gives me a way to try new applications like Splunk without jeopardizing a system and with investing in another box.

The drawback to being able to create new virtual machines on a whim is administering them once they are created. I’ve written a couple of applications that would like to install on all my virtual machines and that is where packaging them as deb files and creating a repository server that apt-get is aware of comes in very handy.

Create a DEB file

The ultimate goal is to be able to type from any of my VMs and have it install properly. All apt-get really does is check all its repository servers for a deb file that contains the “myapp” package, downloads the deb file and installs it. So before we build our repository server (also a VM BTW) we have to create the deb file.

To create the deb file, you need to:

Create the directory structure
Create the control file
Write the install and uninstall scripts
Build the package

Create the Directory Structure

The directory structure of your package will directly mimic the directory structure of the system, with the files in place. For example, if my application requires placing the script A.sh in /usr/sbin and B in /etc/init.d, my directory structure would like:

Create the Control File

Next, we need to create the control file. The control file is placed in a special directory called DEBIAN. After you create that directory your directory structure should look like:

Inside of the DEBIAN directory, create a file named control. Mine looks like:

Most of the entries are self explanatory, but the one you will want to pay close attention to is the “depends” statement. When a user installs your package, if they haven’t installed any of the dependencies, apt-get will allow them to automatically install them.

Install and Uninstall

The next step is to write the install and uninstall scripts. The files in your package will automatically be copied to their corresponding location on disk, but if you need more configuration that has to be performed in your script, you can add that code to a script called “postinst” in the DEBIAN directory. Conversely, you must reverse any changes made during installation in a script called “prerm” which will be called during uninstallation.

For instance, my package contains an xinetd service, so I am going to add lines to “/etc/services”. My “postinst” file will look like:

And my “prerm” file will look like:

Side Note About postinst and prerm

I find sed to be a very useful tool when editing configuration files from a script. This page contains many useful one sed script that may help you write your install/uninstall scripts.

Build the Package

Now our package is complete, we’ve placed our files in the appropriate places and written our control file, install and uninstall scripts. The last step is to package the results into a deb file. To do this, navigate to one level above your package. So if your package is in ~cpetersen/projects/pacakge, navigate to ~cpetersen/projects. From there execute the following command.

That will create the file ~cpetersen/projects/package.deb. You now have a fully functioning deb file. You could install it using dpkg, but we want to use apt-get, so next we’ll look at creating a repository server.

Setup a Repository Server

The repository server is a simple web server (apache in our case) that happens to store the deb files and some meta-data about them. The first step in getting it up and running is installing apache and the dpkg-dev kit which will allows us to create the index.

Next, create the directory where the repository will live. I created a virtual machine to serve as the repository server. Since this is a dedicated machine, I will create my directory directly at the www-root. This would work as well in an apache virtual server. My www-root is /var/www so my deb files will go into /var/www/repository. Finally, you must index the deb files meta data. You can do this with a program called dpkg-scanpackage that comes with the dpkg-dev package. From the directory that you want to server as your repository (for me it is the root /var/www) execute the following commands:

Your repository server is done. If you want more information, I found this page very helpful. Now make sure apache is running and we can move on to the final step, making your clients aware of the server.

Using Your Server

So now you have your deb file, and you have your repository server. How do your client machines know to you use it? apt-get configures its servers using the /etc/apt/sources.list file. So you need to add your new server to that file. I named my server repo01, and I placed my repository in the www-root directory, so I added the following line to all of my clients /etc/apt/sources.list files:

Don’t forget the trailing ” /” (there is a space there). Now update your clients:

And finally you should be able to install your package:

A Note About Adding Packages

Once your server is up and running and your clients are configured to look for it, it is very easy to add or update packages.

Copy the deb file to your repository directory
Re-execute the dpkg-scanpackages command to update the index
Update your clients with sudo apt-get update

Conclusion

Setting up your own repository server and distributing your internal applications as deb files can simplify administration of numberous machines. There are three basic steps, each of which we outlined in this post:

Create the deb file
Setup the repository server
Configure the clients to use your repository

You should now have a working deb file that you can install using apt-get.

ActiveUpload - Easy File Uploads in Rails

Fri, 06 Jul 2007 00:00:00 -0700

Introduction

Our goal was to create a Rails plugin that allows the user to upload multiple files to a website, quickly and easily. This plugin should also give the developer control over what is uploaded, such as maximum filesize allowed, and what filetypes are acceptable. To accomplish this goal, we created the ActiveUpload project.

After researching some alternatives, we settled on a Flash application called SWFUpload. SWFUpload is a Flash movie that was written using FlashDevelop. It is a Flash movie that runs on the client, and handles the client end of the uploads. However, the movie itself is invisible, it is styled with HTML and works via a series of JavaScript callbacks. It is simple yet flexible for the developer, and the user interface has all the functionality we require.

SWFUpload is licensed under the MIT license and Flash Develop is free. For a demonstration of SWFUpload, visit their site.

Design

We need to design a Rails plugin that integrates SWFUpload with a Rails application. Our plugin will consist of three parts.

A Model for storing the files meta data in the database.
A Controller for accepting the uploaded files and manipulating the model.
A FormBuilder extension for rendering the User Interface.

The Model

We started with the model. We knew we would need to store basic information about the file, such as filename and size. We also knew that we would want to attach this model to other models. We decided on a single table, single model design, which stores the filename and size, and attaches to other models using a Polymorphic Association. The migration script looks like:

and the model looks like:

The Controller

Next we decided to tackle the controller. The attachments controller has to do three things:

Upload the file
Create the associated entry in the database
Allow a user to download the file

After many attempts at getting the controller to do this, we settled on a controller with three methods.

create
upload
show

You are probably wondering why create and upload are separate methods. The short answer is, if you create the attachment model inside of upload, there is no way to get the newly created id back out to the web page where you will be creating (or updating) one of your models that has attachments. The long answer, well if you want the long answer, read the “Modifying SWFUpload” section.

It is remarkably easy to accept a file upload inside of rails, all you have to do is:

The full controller looks like:

The FormBuilder

So now we have a model to store the uploaded files meta data and we have a controller to accept the uploaded file and manage the model. Now we need a way of displaying the UI, that is the job of the FormBuilder.

The FormBuilder is a little too big to show here, but I encourage you to look it over, you can find it here.

Basically the FormBuilder write the HTML for you, that HTML includes:

The hidden input containing all the ids of the attachment models associated with this model
The visible list of uploaded files (with delete javascript)
The JavaScript to display SWFUpload

The most complicated part of the helper is the JavaScript to display the Flash application, it looks like:

]]>

That’s it, we now have all the pieces we need to accept uploaded files and attach them to any model. Normally I would jump directly into how you use it however, this project had a particularly interesting side note that I want to delve into. Please feel free to skip to the “Using ActiveUpload” section if you just want to get moving.

Modifying SWFUpload

As I alluded to in “The Controller” there are some limitation within SWFUpload. Ideally, SWFUpload would call its upload script and that would be it, that script would take care of creating the model, uploading the file and associating the file to the model you are actually working on. However, since the files are uploaded before you save the model that is impossible. The next best thing would be for the upload script to create the model and upload the file, then return the newly created id to the web page for associating when you save your model. Unfortunately, there is no way to get information from the upload script back into the page.

Luckily, SWFUpload has numerous callbacks. We settled on a method by which we create the model in the call back that is called directly before you upload the file, the function is called uploadFileStart. This works great, except there is no way (that I could find) to get information from that callback into the upload script. Essentially, we could either associate the file with the attachment model, or the attachment model with your model, but not both.

The solution was to modify the way SWFUpload works. As I mentioned in the Introduction, SWFUpload is open source and FlashDevelop is free, so anyone can modify the source.

We examined the code and found where the call back was called:

You can see the ExternalInterface is what allows SWFUpload to call JavaScript functions. We changed it to the following:

We wanted to keep the solution as generic as possible, so we simply took the result of the callback (if any) and appended it to upload script. So we created a JavaScript call back that returned “?id=XXX” and we were able to get the id of the attachment model that was created in the call back into the upload script. The call back looks like:

Using ActiveUpload

There are six fast and easy steps to using ActiveUpload.

First install the plugin:

Second, generate the migration, model and controller, and migrate your database:

Third, add the following lines to any model you wish to attach files to:

Fourth, add the form helpers to your new and edit views:

]]>

Fifth, add the view helper to your show view:

]]>

Sixth and lastly, add the JavaScript and Stylesheets to your layout:

You should now be able to upload files to your Rails application.

ActiveScaffold

ActiveUpload works well with ActiveScaffold. To get ActiveScaffold to render the correct upload forms you must create a partial form override. Luckily, ActiveUpload comes with one. In the plugin’s public directory ($PROJECT/vendor/plugins/activeupload/public/) you will find a file named _attachment_id_form_column.rhtml. Copy that file to your models views directory ($PROJECT/app/views/$MODEL/).

Since attachment_id is a virtual column, you will need to tell ActiveScaffold to use it, you can do that by adding the following code to your model’s controller:

Finally, you should add the following lines to your attachments_controller.rb:

Notes

You will want to periodically cleanse your Attachments table and the associated files. There are two different ways entries can get stuck in there.

A user adds attachments, then fails to save thier model
A user disassociates an attachment with a model

In both these cases entries are made in the table and files are uploaded. Unfortunately, they are completely inaccessible. You can delete any entries from your attachments table that have a empty attachable_id and attachable_type columns. Don’t forget to delete the associated files!

Additionally, since create and upload are seperate methods, it is theoretically possible to create an entry in the table and not upload the corresponding file. I’ve set SWFUpload to automatically upload (as opposed to creating a queue) so I’ve never seen it happen, but it is possible.

ActiveAuthentication - Active Directory Authentication in Rails

Mon, 18 Jun 2007 00:00:00 -0700

Introduction

In addition to managing our public presence on the web, we spend a considerable amount of time developing internal applications and services. Ruby on Rails makes creating generic webapps a snap; however, for any practical application you need more than just simple CRUD.

How about Authentication?

Most businesses out there, in no small part due to the inflexibility of Windows authentication schemes, reluctantly use Active Directory to manage their user database. Ill conceived parallel authentication schemes that do not fully integrate with AD risk a host of problems: multiple points of failure, more complex security testing, user confusion, complex password management, etc… Instead, wouldn’t it be great if your Rails webapp played nice with AD?

Development

To solve these probles, we created ActiveAuthentication. ActiveAuthentication is a Rails plugin that allows users to authenticate against Active Directory

Requirements

Use same username as password as AD account
Transparently manage the creation of new users to your webapp
Respect the suspension/deletion of users from the AD domain

What’s Out There

Search first, code second. There are a lot of existing authentication plugins for Ruby on Rails. We didn’t want to rewrite what was already working, so after a brief review we elected to build on acts_as_authenticated for our Active Directory authentication system.

Development

With the majority of the work done by others, we made as few modifications as possible. We modified an the Authenticator object by adding and initializing the attributes we need to connect to the Domain Controller. Next we created a new authenticate method.

The authenticate method does the following:

Search the database for a user with the same username as the person trying to login.
Connect to the domain controller with the given username and password.
If step 2 failed, reject the login request, otherwise proceed.
Load the userï¿½s information from active directory.
If you found a user in step one, proceed, otherwise create a new user.
Populate your user model with the information from Active Directory.
Save the new or updated user.

The actual code is as follows:

If you read the code carefully, you probably noticed the active_directory.yml file that is loaded in the initialization method. This file contains all the information necessary to connect to your domain controller. The file looks like:

Simply replace domaincontroller with the name of your domain controller, and yourcompany with your domain name, and you should be good to go. Depending on how your directory is setup, you may have to experiment with the dn, but this setup worked for us.

Installation

The instructions for using the plugin are the same as they were for the acts_as_authenticated plugin.

Update

You also need to install the ruby ldap libraries. On Ubuntu you can do this with the following command:

Then you install the plugin:

Next, generate your user model and update the database:

Now include the system in your application.rb file by adding the following code:

Add the before filter to any controllers you want to secure:

Lastly, make sure you edit the active_directory.yml file in the config directory to match your environment.

That’s it, now you can forget about administering users in your internal applications.

ActiveCalendar - Javascript Calendar on Rails

Sat, 16 Jun 2007 00:50:00 -0700

Background

When I first started with Rails I was amazed at how quickly you can get a project up and running. Coming from a Java background I was used to configuring either Hibernate, then Spring or spending my days tweaking EJB interfaces. The scaffolding in Rails blew me away. However, the basic date and date time renderer, as I’m sure you will agree, left a little something to be desired. It’s easy to change them after the scaffold was created, but that gets old quick. To solve these problems and others, I created ActiveCalendar.

Development

The first part of the process entailed choosing the look of the date renderer. I’ve always liked the DHTML / JavaScript Calendar from Dynarch. Since it is licensed under the LGPL, it was a suitable choice.

Next, I had to write the plugin. I didn’t want a generator because this plugin is designed to be a drop in replacement for the date renderer. It’s not something that has to be generated per scaffold, or configured per field, so I used rails to generate a basic plugin.

Next, I added a public directory where I stored the necessary images and JavaScript and removed all the unused directories and files. In the end I had the following directory structure:

After creating the directory structure, it was time to open the hood and have a look at how rails renders fields. After a little digging, I found the FormHelper module in the following file:

This quickly led me to the DateHelper in:

Inside of the DateHelper module are methods called date_select and datetime_select, which correspond the methods used in the generated new.rhtml and edit.rhtml files. It should be noted that in the first iteration of the plugin, this is where the digging stopped. I overloaded the methods above with my own that rendered the Dynarch calendar. It worked wonderfully, until I discovered ActiveScaffold.

ActiveScaffold is a great plugin that replaces the standard rails CRUD pages with a set of sexy AJAX powered pages. However, it uses a different method to render dates, so it was back to drop downs for me. Since I wanted a drop in replacement that would work with both standard Rails and ActiveScaffold, I was forced to dig deeper into the DateHelper.

It turns out that the DateHelper, like the other FormHelpers, delegates calls to the InstanceTag class. In the final version of the plugin, I overloaded the to_date_select_tag and to_datetime_select_tag methods in InstanceTag and had them call my own DepotDateHelper class. ActiveScaffold must use the same InstanceTag class because once I did that, the calendar began rendering in ActiveScaffold as well.

For good measure, I overloaded the date_select and datetime_select methods of DateHelper and continued to delegate to InstanceTag, but I added a couple of my own options.

Installation

You can install the plugin with the following command:

Next add the appropriate javascript and stylesheets to your layout:

That’s it, your dates should now render as JavaScript calendars.