Bundler is awesome. But Bundler also sucks.
I have been trying to devise a strategy for deploying pre-packaged addons to a Ruby program, but have concluded, much to my surprise, that there is just no simple way to do this in the current Ruby ecosystem. My needs are simple:
- Deliver my addon code
- Deliver any necessary dependencies with my addon code
- Invoke this addon code from a main program without additional a-priori configuration of the main program
The first stop is RubyGems. Gems are great. They are the quantum of dependency management in Ruby, and are analogous to Maven artifacts in the Java ecosystem. An add-on is nominally equivalent to a gem. The tricky part however is obtaining all the gem dependencies. Unfortunately RubyGems only resolves dependencies during installation, so is not helpful in, say, bundling gems. If we are to build our add-on as a gem, then that implies that we need to explicitly run a “gem install” command at deploy time, and we get whatever version of gem dependencies rubygems resolves at deploy time (also assuming the installation machine has network access). This is not desirable. Instead we would like to bundle all the necessary dependencies together with the addon. This is where, you guessed it, Bundler comes in.
Bundler’s mandate is to perform dependency resolution and bundling, for applications. The distinction between applications and libraries is very explicit, in contrast with Maven for example, in which dependency management is handled uniformly (applications are “packaged” the same way as libraries). This is unfortunate and means that one must use the gem specification to specify library dependencies, and Bundler Gemfile to specify application dependencies. Bundler can parse dependencies from gem specs however. Bundler cannot (yet?) handle nesting/composing Gemfiles (because it is only designed for top-level applications). But with some subterfuge we can write something like this:
source “http://rubygems.org”gem “json_pure”Dir.glob(File.dirname(__FILE__) + ‘/ext/*/Gemfile’) do |file|# evileval File.new(file).readend
Here we have coerced our main Gemfile to load the contents of subordinate Gemfiles dynamically. We can reconfigure our addon such that it is no longer a gem, but instead specifies its dependencies in its own Gemfile.
We cannot pat ourselves on the back however, because this cleverness only gets us so far (not that far). Bundler resolves dependencies statically. It requires the static generation of a Gemfile.lock file which contains the static graph of dependencies resolved at the time bundle install was run (even if all the dependencies are already present in the right locations). So this is a dead end. We cannot just drop down an addon with its own Gemfile and dependencies without statically re-resolving the entire dependency graph (see for more details).
What have others done in this situation? There are of course Ruby applications that support “plugins” of some sort. The first one that came to mind was RedCar. It turns out RedCar plugins typically include the source of any dependencies. That is definitely one solution, however I would like to avoid that if possible. It’s possible to “include” third party Git repos in your own Git repo via the submodule mechanism. Unfortunately (to my knowledge) submodules do not support checking out specific paths. Using submodules would require additional mechanisms to filter in/out the right repository content when packaging the plugin.
At this point I’m thinking a simpler solution based directly on RubyGems alone might work. It should be possible to manually resolve dependencies at package time, and use a different mechanism to load them up at runtime.
It looks like a project called Isolate might be the right fit:
Update: I converted the project over to Isolate, and it appears to be working fine. Addon gems can be dropped right into the main program vendor location, and they will be found by the main Isolate.now! invocation, with no qualms.