An indefinitive guide to Composer in Drupal 8

Error message

The spam filter installed on this site is currently unavailable. Per site policy, we are unable to accept new submissions until that problem is resolved. Please try resubmitting the form in a couple of minutes.
Geoffrey

In the past couple of posts, we’ve put several different facets of Drupal 8 under the microscope and discussed how each one is going to change the way you develop for Drupal in the very near future. Now, we’d like to ask a question about Drupal 8 development, one that we believe is going to be quite important in particular for developers who’ve worked within the wider PHP ecosystem.

As we already know, Drupal 8 contains Composer as part of its main distribution. One of the main components of Composer’s workflow is that it processes a central composer.json file to obtain and download a list of dependencies, that are in turn stored in a central vendor directory, and that are in turn loaded via a central ClassLoader object.

This method works fine for Drupal core, as we know what dependencies it’s going to require and it installs everything in a consistent fashion that anything else within Drupal can get hold of.

What we’d like to know is how modules can add their own composer dependencies without getting in the way of Drupal’s dependencies. Could this be done with a custom composer.json file for each module?

To recap: What is Composer?

Composer is a dependency manager. It’s not like a package manager, which downloads a specific version of a specific package and applies it throughout the system. It handles packages on a local basis, different for each individual project it works with.

In this sense, it resembles Ruby’s Bundler gem much more closely than it resembles traditional package managers like apt, yum, MacPorts, Homebrew, or PEAR.

Why is Composer in Drupal 8?

As part of Drupal 8’s effort to embrace contributions from the wider PHP community, it needs a stable way of working with outside libraries in a fashion that doesn’t interfere with that of other systems. Given the way that Composer works on a strictly local basis, it fits Drupal 8’s need well.

However, contributed modules might in turn have their own dependencies that aren’t covered by Drupal 8’s default dependencies, and there doesn’t seem to be a canonical way to set dependencies for individual modules. This is further confused by the fact that Drupal has already had a way of resolving dependencies between modules themselves for some time, but these library dependencies cannot be resolved by Drupal or Composer on their own.

This is the situation we’re currently puzzling over, and this is going to become an increasingly big deal as Drupal 8 gets closer to completion, and module developers start needing to port modules to Drupal 8, or build completely new ones. We're not the only ones trying to work this problem out, as you can see in this list of Drupal core issues:

Method 1: Separate composer.json file per module

One possible solution would be to have each module contain their own composer.json file, specifying their own dependencies independently. You could then install the dependencies when the module is enabled for the first time, like so:

use Symfony\Component\Process\Process;
function mymodule_install() {
  $cmd = 'cd ' . __DIR__ . ' && composer install';
  $process = new Process($cmd);
  $process->setTimeout(3600);
  $process->run();
}

As you can see, we’re using Symfony’s Process component to run Composer from the command line at install time. This particular example depends on a global install of Composer, so you may need to modify it to use a downloaded instance of composer.phar in order to account for all possible cases.

The problem with this approach is that Composer is not built to work the way we’re discussing here. Composer requires a single, unified vendor directory which all dependencies get dumped into, and doesn’t cope very well with having a big global project that uses Composer, which also contains a lot of independently contributed modules that also use Composer. If a Drupal project did use a set of modules that also used Composer, it won’t build it to wrap all those dependencies up in the one vendor directory, and won’t be able to load the class files from the same place.

As a result of this, since Drupal 8 uses the ClassLoader object that gets generated by Composer to load Drupal’s default dependencies, the global ClassLoader won’t contain the class paths that are relevant to your module. You can decide to use two ClassLoader objects, one global, one for your module, but if you decide to try and use both ClassLoader objects at the same time, you’re likely to get some nasty namespace clashes between both objects. Also, if you decide to use several modules that all use Composer in this fashion, you’re likely to have major problems managing all the different ClassLoader objects.

One possible way around this would be to define your own object that sets up the paths required for autoloading, but then the benefits of using ClassLoader objects is made redundant, and in a particularly pointless fashion at that. Alternatively, you could just hardcode the paths to your module’s vendor directory into your module, but that also misses the point of having autoloading functionality.

Method 2: Update root composer.json

Another possible solution would be to update the root composer.json file with your own dependencies. While it makes sense to take due care and update it yourself, it might be more user-friendly to have your contributed module update the root composer.json file and re-run Composer to import any dependencies. This solves the problems from Method 1, and neatly makes your module’s dependencies available throughout your Drupal instance.

This could be done in Drupal 8 using a method like this:

 use Symfony\Component\Process\Process;
 function mymodule_install() {
   $composer_loc = DRUPAL_ROOT . '/composer.json';
   $composer_data = json_decode(file_get_contents($composer_loc));

   // Adding dependencies and rewriting composer json
   $composer_data['require']['foo/bar'] = "1.0.0";
   file_put_contents($composer_loc, json_encode($composer_data));

   $cmd = 'cd ' . DRUPAL_ROOT . ' && composer install';
   $process = new Process($cmd);
   $process->setTimeout(3600);
   $process->run();
 }

(Note that this method depends on DRUPAL_ROOT, the behaviour of which may end up changing in Drupal 8. Always follow the API documentation when trying to confirm these things.)

However, there are also problems with this method.

Let’s propose a hypothetical situation in which we have two contributed modules on hand, and we need to use them both. Both rely on an external dependency... in this case, let’s say it’s Symfony’s DOM Crawler. Module A is in active development and has been updated recently, and relies on symfony/dom-crawler version 2.3.x-dev at time of writing. Module B, on the other hand, is pretty stable and hasn’t required a major update in some time, but requires v2.2.1 due to some particular feature of that version that doesn’t yet have an equivalent in v2.3.

What happens when you install them both? It’s likely that somewhere along the path, there’ll be a major namespace clash and one of the modules will end up using a dependency that is completely incorrect for it. As you can see, this is a major problem with this approach; this is the sort of issue developers and site builders would need to resolve manually, even to the point of consultation with the developers of the libraries and modules involved.

Furthermore, there’s a problem that might cause difficulties once Drupal 8 has a stable release and is used by the general public. If subsequent versions of core update its dependencies, site developers will have to manually update the composer.json file. They’ll have to do this either by patching in core’s changes, or overwriting it and bringing in each contributed module’s dependencies. This leaves the risk that a dependency will be missed.

More severely, this raises the issue of whether or not editing composer.json in any way counts as hacking core. As has been drilled into our heads over and over, hacking core harms wildlife and wrecks the developer experience for yourself and others.

Some developers have proposed that Drupal core should become a dependency of Drupal, and let Composer handle all dependency loading when Drupal is installed. This would allow Drupal to ship with a global composer.json file that encompasses the entire project, distinct from Drupal core’s dependencies. This sounds like a pretty good idea, but in the weird event that a module requires a version of a particular library that clashes with one that Drupal core is using, this may cause problems.

Finally, handling dependencies on your own as part of a module’s install procedure is not very maintainable, and is kind of hacky to be honest. It’d be a much better idea to contribute back to the discussion surrounding Drupal core, to try and find a solution to this problem. Perhaps there’s a way to integrate this into core in a fashion that everyone can make use of.

Method 3: Stop worrying and love Symfony

There is one more, particularly radical method that might help resolve some of these issues, and it follows on from the previous one.

There’s already been some discussion about using composer.json as the replacement for .info files in future versions of Drupal, so this might be one place to start. Since Drupal 8’s developers have gone to great lengths to shed the antiquated NIH (Not Invented Here) mentality in favour of PIE (Proudly Invented Elsewhere), it might make sense down the line to start postulating the idea of Drupal modules effectively being standalone Symfony-compatible components that have Drupal as a dependency in turn. This would mean that they would be listed on a service like Packagist, and integrated into a given project using Composer.

The biggest benefit of this method is that the issues we mentioned above, such as version clashes, difficulty of managing multiple vendor directories and ClassLoaders, diminish in importance or vanish completely. However this wouldn’t solve every problem with dependencies – as long as there are different components of any project that need to work together, there will almost always be collissions and arguments. In terms of admin experience, it would be interesting to see how this would work for ordinary site builders.

Also, any move like this is not likely to take place any time soon. Drupal 8 has undergone feature freeze already, and while there has already been some preliminary planning for Drupal 9, it’s unclear that a move to a more Symfony-esque style of module packaging is set to take place.

However, this opens up new possibilities for Drupal modules that may extend beyond Drupal itself. There are already packages out there like Bootstrap that don't necessarily need Symfony, but which do ship with composer.json files. It could even be possible for Drupal modules to be written such that their classes could be used without running in Drupal core... However, it's probably way too early to start thinking about this.

The development/administration experience

It could be that we could use community contributions to improve the way that module dependencies are handled. The Composer module is a good example of one possible approach, which allows site builders to download and install dependencies from the command line.

However, the Composer module depends heavily on prior knowledge of Drush and the command line, which isn’t very friendly from the standpoint of a site builder, webmaster or end user. Also, using a contributed module to handle functionality that really should be in core isn’t a very good situation, as we’ve already experienced with modules such as CCK, Views and so on.

Awaiting resolution

We’re aware that this is a topic of ongoing discussion, but we don’t yet know if there has been any consensus reached or any concerted push behind researching a solution. We’d be most interested in seeing something like Method 2 worked into core, so that all module developers can make use of it, but the idea of using a command line component for core functionality has some developers concerned.

If anyone knows how to do it, or can think of a better way to do it than what we’ve discussed here, we are listening and would love to hear from you. If there is anything we could possibly do to help find a solution to this issue, we'd be most interested in lending a hand.

Here’s what we do know

If you'd like to learn more about Drupal 8 and how it works with Symfony, you can find us in DrupalCon Portland this May, where we'll be giving a training session on Drupal 8 and Symfony. Registration is open, and tickets are going fast, so get in quick!

Comments

Man, I kinda wish I wasn't doing training so I could come sit in yours. I might have to find me a sub. ; )

Has everyone watched http://munich2012.drupal.org/content/drupal-has-dependencies-lets-manage-them where the author of Composer talks about Composer and drupal 8 using it (and that Composer had been oddly placed in a subdirectory of the project).
I prefer his solution, move it up all the way to the root folder and have a global composer with like drupal as a dependancy. I say treat it like .htaccess -- its just configuration of the application in .json structure. Hacking .htaccess is not hacking core. Screwing up .htaccess can also horribly ruin your day ...

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

glqxz9283 sfy39587p10 mnesdcuix7