Out of Copenhagen

Hop 0.0.1 released

by jacob on Feb.05, 2011, under Uncategorized

I often find myself googling the same pages over and over and over. And usually I’m working on the command line when I suddenly need to lookup a piece of information, that I’ve looked up many times before. Basically I end up using Google to get to the same pages over and over.

There’s two things that bothers me about this workflow:

  • I’m often on the command line when I need to find a given piece of information. Then I have to shift to my browser.
  • I’m using Google to search for stuff I already know where to find. This introduces an unnecessary step.

When I find myself doing repetitive tasks like this I try to think of easier and faster ways of doing it. And so I decided to build Hop. So now I can type:

hop md.links

And my browser will open up the syntax page for Markdown and scroll to the links section.

If you want the goodness of Hop installing it is as easy as ‘gem install hop’ - provided you have a unixy system with Rubygems installed. For further details checkout the Hop readme. I hope you will find Hop to be a useful tool.

Leave a Comment more...

Packaging changed files with Git

by jacob on Jul.24, 2009, under git

A non-git using client of mine requested that I deliver all changes to the codebase I’m working on as a tar-ball with only the changed files since the last delivery. I’m thinking this might be somewhat common in the Windows environments out there. So to ease the job I made a little script that would extract all files changed between two revisions and package it up nicely. The script looks like:

It can be run with “script-name old-tag new-tag” - where old-tag and new-tag should be either tag-names or branch-names or something else that git show-ref understands. If there’s a better way to do this please let me know.

The script will create a new tar-ball in the pkgs subdirectory of the current directory. The pkgs directory should be created before running the script.

Example:

I send the client the full codebase at revision 123 and tag it as release1. Then the client comes in with another feature request which I implement through a series of commits touching files and b and finally when I’m done I tag the new commit as release2. By using the script with release1 and release2 as arguments I will get a tar-ball containing the files a and b as they look at release2.

Comments Off more...

The Dead Method Eliminator is here

by jacob on Jul.20, 2009, under Uncategorized

Ever wondered how many of the methods in the Ruby on Rails project you’re currently looking at are actually called from within your program? Yeah, me too. So I wrote up a very quick little static analysis tool to tell me. Here it is:

Beware - it’s static. So it will only tell you if there’s no calls anywhere in the code for a given method name. Meaning there are a few caveats:

  • Two methods on different classes with the same name might be incorrectly detected as being in use.
  • If a method is called from an unreachable part of the code it will be incorrectly detected as being in use.

If you know of any better tools for doing this please let me know in the comments.

1 Comment more...

On repeatability

by jacob on Jun.04, 2009, under meta

Repeatability: To be able to do exactly the same thing again and again.

I would like to talk a bit about repeatability in the context of computers and software. Repeatability in this context means many things to me. Examples of things that could be made repeatable are:

  • Given a CSV file, remove a given column, validate another column, put it into a server
  • Installing a webserver
  • Calling a webservice and graphing the result
  • Setting up a development environment

For each of these examples repeatability means the ability to do them again and again and arrive at the exact same result each time. The contrast to repeatability is to do the same thing over and over and arriving at different results each time. An example of a process that would not be repeatable would be to ask a human to validate the column in the CSV file mentioned above by hand. At least not if the CSV file is sufficiently large. In that case human error would most likely mean a different output each time.

There is a concept closely related to repeatability known to computer programmers, namely the concept of idempotency. Idempotency means that a given operation can be executed multiple times and the result will be the same as if the operation was only executed one time. This means that for something to be idempotent it must be repeatable, whereas the opposite does not hold: An operation can be repeatable without being idempotent. An example of an operation that is repeatable but not idempotent is creating a file with a filename matching the time of creation.

Idempotency is an interesting concept on it’s own but I will not delve further into it in this post. Suffice it to say that thinking of idempotency might yield benefits when thinking about repeatability.

Now that I have explained what I mean by repeatability, I would like to explain why I find repeatability an important and valuable concept to apply in practice.

First and foremost is automation. Before we can claim to have a repeatable operation the operation must be defined so well, that it’s possible to get a machine to do it. Every step must be so well defined, that there is no room open for interpretation. If not we risk different interpretations leading to different results. As the process is so well defined it becomes easy to convert into a software program (This should be taken in the broadest sense possible). And once it is available as a software program the process can be repeated indefinitely with very little cost.

The second gain from repeatability is the capability to fix errors in the process also known as debugging. By having a repeatable and well defined process we can start to analyze the individual steps of the process and correct any steps that might be leading up to an incorrect result. Without a repeatable process we would simply have to do the process again, but even if we was able to validate that the result was correct on the second run, we wouldn’t know if it was due to a process improvement or just an artifact caused by us doing something a little different.

A consequence of the combination of automation and debugging is that it becomes much easier to perform the process once more should a problem be uncovered and thereby saving time by not having to manually do the work again.

As repeatability is an interesting concept on many levels, from basic software installation to small programs doing something as simple as filtering a CSV file there’s a lot of tools providing some of the benefits on these different levels. There are a few tools that are mostly universally applicable when working with repeatability in mind.

The first is a version control system. This allows us to be repeatable in the programs developed for repeatability. Yes, feel the meta. Version control systems are universally known among programmers these days, so there’s not much reason to discuss them further.

The second category of tools is what is called build tools. These exists in a multitude of variants, from old-school Makefiles to Ant files. A build tool is not just for building though, it is usually well suited for automating any kind of repeatable operation. So instead of writing a full program for any given operation use the build tool to help as much as possible. Build tools have some features that can be very helpful when setting up a repeatable process, but that discussion will have to wait for another post.

Leave a Comment more...

Using multiple repositories as a base with git

by jacob on Jun.03, 2009, under git

When developing you sometimes need to include stuff from multiple sources, say plugins, vendor material or other things. Depending on which version control system you are using there are various ways to go about this. In this post I will focus on a special case and I will be using git as my VCS. The technique might be applicable to other distributed VCS’ but probably not to any centralized VCS (Subversion f.ex.).

I will be talking about the special case where you want to merge two repositories into the same root folder. Say you have two repositories that looks like:

And:

What this technique will allow you to do is to merge the two repositories so that files A1, A2 and B1 will be in the same root folder and it will have two subfolders dirA1 and dirB1 with the respective files. That means you won’t be able to pull a repository into vendor/plugins with this technique, for that you will have to resort to other ways.

Now, when would you want to do this? I tend to do it if I have a base template for a given project type I want to use across multiple projects and I want to be able to easily update the various projects when I change the base template. An example could be a Rails project where I have a base Rails project with some user authentication and I want to use this as a starting point for multiple projects.

The technique in itself is very simple and is based on the fact that git is a distributed VCS which can use more than one repository as a root. The implementation of the technique goes like this. First I setup my base project:

Then I create my new project:

Now to use my base project in my new project I can simply pull the base project into the new project:

And then I have the files from my base project in my new project. Should I later on do any changes in my base project and want to update my new project all I have to do is pull from the base project once more.

One thing I tend to do when applying this technique is to add the base project as a remote in the new project and then do a branch based of the base project to do any local changes to the base functionality in there:

By having the base project as a branch in the new project it becomes easier to develop changes to the base functionality within the new project and then push changes to the base project back to the original base repository.

There is (at least) one problem with this technique though, and that is you will have the code from different repositories mixed up and it might become difficult to figure out where a given piece of code originated and thereby if it should be pushed back to a base project or just be part of the regular project. In my experience this is not too bad as long as the base project is well defined but the technique should probably not be applied indiscriminately.

Comments Off more...

3 small things I like about Ruby on Rails

by jacob on Jun.02, 2009, under rails

Ruby on Rails has been a hotly debated topic the past couple of years. Since the first public release in 2004 it has spread like wild fire in the web development community. Some love it, some hate, some just don’t care. But one thing holds true, it has set the bar for web development frameworks to the point where even corporate giants are using it as inspiration.

What I want to talk about here is not the great things in Rails - there are already plenty of blog posts touting these things - and plenty of blog posts telling us why there’s really nothing special about it, that you could just as well do it in framework X or language Y. No, what I want to talk about is the little things that makes Rails great to me. The little things that don’t get that much attention, the things that are just there, which seasoned Rails developers probably do not give much thought and outsiders do not really see the value of.

The log

First up there’s the log. Rails by default comes with 3 different log files. One for when you’re doing development, one for when you’re doing testing and one for your production environment. The information in these log files differ from development to production, with the production log having less information compared to the development log.

Here is an example of what a log entry looks like in the development log:

What we see is that the following information is collected and presented to us:

  • The primary method called for responding to a request (the action in Rails lingo).
  • The arguments passed to that action, URL arguments if the request is a get and form encoded arguments if the request is a post.
  • The database queries performed as part of handling the request is shown including the execution time of each query.
  • What templates are rendered as part of the request.
  • The execution time spent completing the request.
  • The HTTP status code returned from the request.

What I use the log for is primarily two things, the first is debugging. Having all the arguments sent by the client readily available is a great help. I can immediately see if I’m getting the data I expect into my application. The second is for performance profiling. If something seems slow I bring up the log to see exactly what is making the application seem that way.

Of course you can also put in any information you want yourself by putting logging statements in your code. Personally I find that is not needed very often for me though.

The log is not the end of the story. There are various services out there that can provide greater insights or make things more manageable both in regards to debugging and profiling. But when you’re just starting out the log is a great tool.

The console

Sometimes when you’re developing your want to try something out or see the output of some method or just poke your application in various ways. For this purpose Rails provides a console which will load up your app and make it readily available to you. Within the console you have access to all your classes and can execute any arbitrary piece of code you can come up with.

Here is an example of a console session:

Once you know that >> is the console prompt and => is the result the example should be self-explanatory.

I tend to use the console for mucking around with my data when I’m in development. Say I want to quickly create 10 tasks or update a bunch of records I don’t go to SQL, I go to the console. If something seems to have gone wrong or the data has somehow become invalid it’s easy to bring up the console and update the data as I want.

Additionally I like to use the console for exploratory programming. If I have a question about syntax or want to prototype an algorithm quickly I tend to use the console to verify that things work as I expect them to.

Rake tasks

I am a big believer in the value of reproducibility. When talking about development one of the things I greatly appreciate is the one-touch-whatever. Meaning that if I want to update my database schema, run my test suite or regenerate my cache I want to be able to do it with one command. I definitely do not want to have to read through a script written in natural language and having to translate it into commands or just enter the commands from the script myself.

For doing this Rails uses the Ruby standard Rake language and runtime system. Here’s a few example of some built in Rake tasks:

As can be seen I can do quite a few things out of the box with Rake tasks. At least some of which are backbone reflexes for seasoned Rails developers.

I can also easily write my own Rake tasks, here’s a quick example to wipe out all Tasks from the database:

So I have full access to the classes of my application and can call methods on the classes without any further ceremony.

The Rake tasks mechanism is useful for routine maintenance jobs like clearing stale data, data exporting and importing and data migration amongst other things. Another thing I like to use Rake tasks for is to communicate with other developers. Instead of writing down a series of steps to be done to perform some action I can write a Rake task and be sure that they will do the steps exactly as intended - full reproducibility. Oh, and did I mention it is a lot more fun to run commands than write documentation?

Conclusion

I’ve presented 3 small things which makes my life as a developer easier. These things are not rocket science. They’re not something you can’t do with whatever other technology you might prefer. The thing I like about Rails is that they come for free. They are readily available to every Rails developer out there, no installation, no configuration needed. To me this means I don’t have to spend time getting these things working, I can spend my time working on the meat instead of the boilerplate delivering value to my client.

Comments Off more...

Capistrano deployment with git failing periodically

by jacob on May.24, 2009, under capistrano

If you’re using Capistrano to deploy from a git repository on OS X with the copy strategy and a copy cache you might be running into a problem where every now and then your deployments will end up not having any files in the deployed directory. One reason for this is that the files from the temporary directory holding the copy cache is wiped, that is the files are, not the directories. If you experience this problem look for something like the following output from Capistrano:

  * executing `deploy'
  * executing `deploy:update'
 ** transaction: start
  * executing `deploy:update_code'
    executing locally: "git ls-remote http://github.com/foo/bar master"
  * refreshing local cache to revision a9eaabd7656a59594dcc2f855c36906e3d857a03
      at /var/folders/+d/+dQslZbbElaSoZEpfVoFdk+++TI/-Tmp-/myproject
executing locally: cd /var/folders/+d/+dQslZbbElaSoZEpfVoFdk+++TI/-Tmp-/myproject &&
  git fetch -q origin && git reset -q --hard a9eaabd7656a59594dcc2f855c36906e3d857a03
fatal: Not a git repository (or any of the parent directories): .git

The important thing to notice is the last line. Git is telling us that our cache does not contain a git repository anymore.

I am assuming that it is OS X that somehow cleans out the tmp files every now and then, though I’m not sure that is actually the case. If it is indeed the case then a solution to the problem would be to have OS X remove directories alongside the files in those directories.

Another solution is to set the copy_cache in the deployment recipe to a full path instead of true (in which case Capistrano creates a regular operating system level tmp dir). F.ex: set copy_cache, “/tmp/deploy-cache/myproject”. By doing this we’ve hopefully placed the copy cache in a place where it will not be wiped and so we shouldn’t experience the issue anymore.

3 Comments more...

Quickly manage hostnames on OS X

by jacob on Apr.12, 2009, under Uncategorized

Sometimes it’s nice to add domain names to your local machine even if they don’t exist in the real domain system. F.ex. you could be deploying some stuff and the domain is yet to come online but you want to reach the server anyways or you’re developing a web app on your local machine and need different domains to call your app on.

One way of handling this problem is to edit /etc/hosts and adding the lines you want in there. There is another way on OSX that lends itself better to automation and scripting: The small dscl utility. With this adding and removing domains becomes as simple as:

Of course this is a lot to write so you might want to make a little shell script that does it for you, say:

Put this into a file called ~/bin/add-domain and then setting up a new domain pointing to localhost is as simple as: add-domain test.foo.local

I will leave creating the delete-domain script and expanding the add-domain to handle arbitrary IP addresses to you.

2 Comments more...

One touch Rails development environment

by jacob on Mar.29, 2009, under Uncategorized

I’m an automation fan. All those little things that I do over and over and spend time on. I just feel the need, the need to automate. I’m not saying it’s rational. Anyways…

I just did an Applescript to set myself up with a Rails environment the way I like it, that is one iTerm tab for each of the following:

  • script/server
  • script/console
  • tail -f log/development.log
  • shell

Applescript has kind of a weird syntax compared to what I’m used to, so I guess it’ll take me at least 5 years before this particular automation has paid off on itself. Oh well. Here’s the script:


to maximize for a
tell application “Finder”
set _b to bounds of window of desktop
end tell

tell application a
set bounds of front window to _b
end tell

end maximize

on execute(cmd, posixPath)
tell application “iTerm”
set myterm to (the first terminal)
tell myterm
set mysession to (current session)
tell mysession
set name to cmd
write text “cd ” & posixPath
write text cmd
end tell
end tell
end tell
end execute

on executeNew(cmd, posixPath)
tell application “iTerm”
set myterm to (the first terminal)
tell myterm
launch session “DefaultSession”
set mysession to (last session)
tell mysession
set name to cmd
write text “cd ” & posixPath
write text cmd
end tell
end tell
end tell
end execute

to openTermIn(posixPath)
tell application “iTerm”
set myterm to (the first terminal)
tell myterm
launch session “DefaultSession”
set mysession to (last session)
tell mysession
set name to “terminal”
write text “cd ” & posixPath
end tell
end tell
end tell
end openTermIn

on run argv
set posixPath to item 1 of argv
execute(”ruby script/server”, posixPath)
executeNew(”ruby script/console”, posixPath)
executeNew(”tail -f log/development.log”, posixPath)
openTermIn(posixPath)

tell application “TextMate”
open posixPath
end tell
maximize for “Textmate”
end run

—————–

To compile:
osacompile -o startrails.scpt startrails.applescript

—————–

To run:
osascript startrails.scpt /path/to/rails/project

I’ve setup an alias so I can run the script like “re /path/to/rails/project” to make it even easier.

There’s quite a lot of duplication in the three open/execute commands. If you know a better way please leave a comment.

Leave a Comment more...

Capistrano recipe for deploying Wordpress

by jacob on Mar.23, 2009, under capistrano

There’s been a request for a Capistrano recipe for deploying Wordpress. Here it is:

Basically you should be able to just put the recipe into a file name "Capfile" in the root of your project, change the parts with the square brackets around and then you're ready to go.

There’s a few things worth mentioning. First up the recipe assumes you’ll be deploying from a local git repository rooted in the current working directory. Change line 2 if that doesn’t suit you.

Then in line 26 I add write permission to everything for the group. This is to allow wordpress to write different files into the directory if needed. You might need to setup a proper group or you can disable this if you don’t want it.

In line 33 I symlink config/apache.conf to the place where I store my apache configs on my server. This is for easy maintenance of the apache config file. Also if my server should get nuked I can easily install on a new server when I have the apache config versioned. You will need to make a config folder in the root of your project for this to work. I usually have a config folder for config stuff and a public folder for the Wordpress files, just to make sure my config is not readable to everybody.

And that’s all you need to deploy Wordpress with Capistrano.

Update: It seems there’s a project on Github for exactly this. Might be worth checking out too.

Update june 30: The recipe has been modified to handle uploads and the htaccess file. You will need to tell wordpress where to find the uploads directory by modifying Settings -> Miscellaneous -> Store Uploads in This Folder. I point it at the shared/upload directory of the capistrano deployment. Also the htaccess is now symlinked in case the user chooses to modify it from within wordpress (in which case you shouldn’t copy it from source control onto the deployment).

1 Comment :, more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...

Archives

All entries, chronologically...