Composer workflow for developing proprietary Magento 2 extensions

PHP Composer LogoIt appears future came for us Magento developers in the end. Magento 2 platform is alive and kicking, and with it modern PHP tools like Composer entered our daily routine. And it's so much easier now the Composer is here to take care of my dependencies! Khm. Not so fast. To be honest, packaging software into neat fully decoupled packages is still a utopia. We model tightly coupled reality which is not something you can hide behind package manager. Nevertheless, there are some neat concepts behind Composer and Magento 2 working together, lets check it out.

Starting out with Magento 2 and Composer, your first question might be: OK, but how do I develop and restrict access to my proprietary Magento 2 extensions? Unfortunately there isn't much documentation on this topic, so this might not be the official way of doing things, but it works pretty well for us. So the goal is to set up system allowing us to securely develop in-house Magento 2 extensions shared amongst multiple Magento installations, each inside separate private Git repository.

Composer repositories

When you do composer require whatever/whatever:xyz, Composer simply looks up this package inside all the Composer repositories your project's "master" composer.json has configured, and pulls the information on where to find this particular package as instructed by the repository. An excerpt from stock Magento 2 composer.json reveals your Magento 2 installation will by default look things up in Composer repository hosted at https://repo.magento.com/:

.
.
.
"license": [
    "OSL-3.0",
    "AFL-3.0"
],
"repositories": [
    {
        "type": "composer",
        "url": "https://repo.magento.com/"
    }
],
"require": {
    "php": "~5.5.0|~5.6.0|~7.0.0",
.
.
.

If you are developing proprietary Magento 2 extensions, an idea is to have your own repository up there right below https://repo.magento.com/. Simple approach is to add VCS type repository for each and every proprietary (or not) extension you require on your project:

.
.
.
"license": [
    "OSL-3.0",
    "AFL-3.0"
],
"repositories": [
    {
        "type": "composer",
        "url": "https://repo.magento.com/"
    },
    {
        "type": "vcs",
        "url": "git@github.com:DevGenii/Awesomness.git"
    }
],
"require": {
    "php": "~5.5.0|~5.6.0|~7.0.0",
.
.
.

This works but it's kind of ugly because you have to add these repositories to composer.json separately for each and every Magento 2 project requiring these extensions. And what if you decide to switch to Bitbucket at some point?

Even better approach is to host your own Composer repository, and to add this single repository to every Magento 2 installation requiring your in-house or otherwise private Magento 2 modules, for example:

.
.
.
"license": [
    "OSL-3.0",
    "AFL-3.0"
],
"repositories": [
    {
        "type": "composer",
        "url": "https://repo.magento.com/"
    },
    {
        "type": "composer",
        "url": "https://repo.devgenii.com"
    }
],
"require": {
    "php": "~5.5.0|~5.6.0|~7.0.0",
.
.
.

Satis

Composer repository itself is simply collection of JSON files hosted remotely containing instructions on how to map Composer packages inside repository to resources like ready-made artifacts, or more importantly - public or private Git repositories hosted pretty much anywhere. And fortunately Composer guys built a tool for creating this Composer repository as well as keeping it up to date as VSC repositories of packages inside change by adding new branches and tags. Tool I'm talking about is Satis. Installing Satis on your own server comes down to cloning Satis repository on your server, creating satis.json file containing list of Git repositories Composer repository will contain, as well as configuring cron job for keeping this list up to date as remote Git repositories changes. Here are a few snippets demonstrating this on example of subdomain repo.devgenii.com serving content over HTTPS using Nginx server.

First thing we need to do is set up public key authentication for Linux user we plan to use for maintaining our Composer repository, because this user needs read access to each of the Git repositories we plan on having in our Composer repository. This is repodevgeniicom user for the purpose of this demonstration, I'll leave you in the capable hands of GitHub documentation concerning deploy keys.

Next we login as repodevgeniicom to our server, and create satis.json template file describing the contents of future Composer repository:

cd /home/repodevgeniicom/
cat << EOF > satis.json
{
    "name": "DevGenii Composer Repository",
    "homepage": "https://repo.devgenii.com",
    "repositories": [
        {
            "type": "vcs",
             "url": "git@github.com:DevGenii/Awesomness.git" 
        }
    ],
    "require-all": true
}
EOF

Feel free to get crazy in the "repositories": [] section by adding all of your Magento 2 modules Git repositories, but keep in mind that Satis needs read only access to these repositories in order to build and maintain its index.

Next we download Composer, set up Satis project and generate Composer repository JSON files out of our /home/repodevgeniicom/satis.json template, and make sure JSON files are placed inside our soon to be document root:

cd /home/repodevgeniicom/
curl -sS https://getcomposer.org/composer.phar -o composer.phar
php -f composer.phar create-project composer/satis --stability=dev --keep-vcs
php -f satis/bin/satis build /home/repodevgeniicom/satis.json /home/repodevgeniicom/public_html --no-interaction

Next here's an example of Nginx configuration file required to serve this Composer repository:

cat << EOF > cat /etc/nginx/sites-available/repodevgeniicom
server {
    listen 443 ssl;
    listen [::]:443 ssl;
 
    server_name repo.devgenii.com;
 
    access_log /var/log/nginx/repo.devgenii.com.443.access.log;
    error_log /var/log/nginx/repo.devgenii.com.443.error.log;
 
    root /home/repodevgeniicom/public_html;
    index index.html index.htm;
 
    include snippets/ssl.conf;
 
    location / {
        try_files   $uri $uri/ =404;
    }
 
}
EOF
ln -s /etc/nginx/sites-available/repodevgeniicom /etc/nginx/sites-enabled/repodevgeniicom
service nginx restart

Lastly, we setup a cron job to update our Composer repository with upstream releases and branches every 10 minutes:

*/10 * * * * php -f $HOME/satis/bin/satis build $HOME/satis.json $HOME/public_html/ --no-interaction >/dev/null

Magento 2 module Git repository

We can not point our satis.json to just any repository, because these must have valid "slave" composer.json file in project root, in order to properly declare Magento 2 module to any Composer repository interested in having this package in its index. An example of proper "slave" composer.json file taken from imaginary devgenii/awesomeness Magento 2 module:

{
  "name": "devgenii/awesomeness",
  "description": "N/A",
  "type": "magento2-module",
  "version": "1.0.0",
  "authors": [
    {
      "name": "Marko Martinović",
      "homepage": "https://devgenii.com"
    }
  ],
  "license": [
    "proprietary"
  ],
  "require": {
    "php": "~5.5.0|~5.6.0|~7.0.0",
    "magento/framework": "~100.0.9"
  },
  "autoload": {
    "files": [
      "src/registration.php"
    ],
    "psr-4": {
      "DevGenii\\Awesomeness\\": "src"
    }
  }
}

Note the magento2-module type. Now this is Git repository Composer repository can work with.

Private Git repository authentication

Since Composer acts simply as intermediary between your Magento 2 project and some remote Git repository out there, private repository access itself is a proper place to enforce any restrictions. More precisely, we simply use public key authentication in order to make sure that doing:

php composer.phar require devgenii/awesomeness:dev-master

dies a horrible death if person doing this action doesn't have access to our private Git repository - the one Composer repository has mapped for devgenii/awesomeness Magento 2 package. One thing to point out if you're relying on public key authentication and use GitHub, Composer guys did an exception there so when Composer sniffs out that GitHub is on the other end, it attempts token based authentication instead of public key. Simple:

php composer.phar require devgenii/awesomeness:dev-master --no-interaction

should cause Composer to attempt public key authentication, even though private Git repository is hosted on GitHub - note the --no-interaction there. And you can use this flag on every Composer command to bypass token based authentication with GitHub.

Local development workflow

Although we used Modman for local development on developer machines at first, having to git-ignore each module symbolic link in app/code proved to be a bit awkward. Therefore we use Composer now there as well, but we instruct it to Git clone our in-house modules instead of pulling in artifacts. The development is then carried out inside each module Git repository inside vendor directory, directory that's ignored by default in host Magento installations .gitignore file. In order to make sure Composer will prefer source when installing our own proprietary Magento 2 modules, developers need to add following to their global Composer configuration:

{
    "config": {
        "preferred-install": {
            "devgenii/*": "source"
        }
    }
}

by editing their ~/.composer/config.json. This will make vendor/devgenii/awesomeness directory a Git repository of it's own after Composer does its magic, which is perfect for developing Magento 2 module separately from host Magento installation specific code. While we're at host installation specific modules, these are developed inside app/code and are versioned inside host Magento 2 installation Git repository.

This pretty much covers Composer workfow we use when developing in-house or proprietary Magento 2 extensions for DevGenii clients. If you have even better approach or thoughts concerning workflow I just outlined, please don't hesitate to share.

DevGenii

A quality focused Magento specialized web development agency. Get in touch!

5 thoughts on “Composer workflow for developing proprietary Magento 2 extensions

  1. Alan Kent

    Thanks for sharing! I had a question – how do you find it easiest to manage your source code while buildings project? Do you put each module into a separate GIT repo? (Then it would install under ‘vendor’.) Or do you put it into the project under app/code? I am wondering best way to have separate GIT repo without making development too painful.

    Reply
    1. Marko Author

      Hi Alan,
      thank you for your question. Each module containing code for which it makes sense to create a package (code not specific to one single project) in its own Git repository, and added to composer.json as a requirement to Magento 2 installations, where code is installed to vendor by composer install. In local development Composer is used as well, but we use –prefer-source param in order to clone instead of downloading ready-made artifact while doing composer require. Code specific to Magento 2 instance is kept inside app/code and is versioned inside Magento 2 instance Git repository.

      One important thing concerning production, we’re not letting Composer near there for now. All the Composer packages are pulled in locally on developer’s machine using proper composer workflow, and are deployed to production server together with the rest of Magento 2 framework and project specific code.

      Hope this answers your question Alan.

      Best regards,
      Marko

      Reply
  2. Matthias Zeis

    Hi Marko,

    thank you for sharing your approach! We’re experimenting with pretty much the same workflow. The concern our devs are having with working inside vendor/ is that they’re afraid of losing work when they forget about uncommited code and do a composer update.

    How are you feeling about this?

    Matthias

    Reply
    1. Marko Author

      Hi Matthias,
      this is valid concern. Luckily Composer will detect commits which are not pushed or local changes when doing composer update:

      https://github.com/composer/composer/blob/master/src/Composer/Downloader/GitDownloader.php#L215-L295

      Even better, there’s https://getcomposer.org/doc/06-config.md#discard-changes parameter allowing us to control this behavior depending on whether we develop locally or configure unattended installs like CI.

      Best,
      Marko

      Reply
      1. Matthias Zeis

        Hi Marko,

        thank you very much for the explanation! I heard about the commit detection (some people don’t trust it, I don’t know why) but didn’t see the discard changes parameter.

        Matthias

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *