Running Magento 2 Tests and Deployments via Bitbucket Pipelines

We continuously try to improve the quality of our Magento projects. A big part of that is running all checks and tests automatically for every change. Since we use Bitbucket for our client projects, we do that via Bitbucket Pipelines. Most of it is transferable to GitHub Actions, GitLab CI or any other CI system.

We will not explain the Bitbucket Pipelines syntax itself. If you want to learn the basics, check out the official documentation.

Overview

Bitbucket Pipelines is configured via a bitbucket-pipelines.yml file in the root of the repository. It defines when a pipeline runs (e.g. for every pull request, or when triggered manually) and what it does: each pipeline consists of steps, and each step runs a list of shell commands in a fresh Docker container. If a command fails, the step fails.

We use it for two things: running all checks and tests for every pull request – nothing gets merged unless they are green – and deploying to the dev/staging and the live system.

In our projects, the pull request pipeline consists of three steps, which run in parallel:

1. Linting, Static Checks, Unit Tests and Dependencies
2. Integration Tests
3. End-to-End Tests Against a Real Client Database

Running steps in parallel does not save build minutes, but you get feedback much faster.

Step 1: Linting, Static Checks, Unit Tests and Dependencies

This step needs no database or other services. First, we install the system packages and PHP extensions Magento requires, raise the memory limit and install Composer. After composer install, we run PHP_CodeSniffer and PHPStan, execute the unit tests, check the module dependencies and finally audit all packages for known security issues.

- step:
    name: Linting & Static Checks & Unit Tests & Module Dependency check
    caches:
      - composer
    script:
      - apt-get update && apt-get install -y git libxslt1-dev libxml2-dev libicu-dev libpng-dev libzip-dev unzip
      - docker-php-ext-install -j$(nproc) bcmath ftp gd intl pdo_mysql soap sockets xsl zip
      - echo 'memory_limit = 3G' >> /usr/local/etc/php/conf.d/docker-php-memlimit.ini
      - export COMPOSER_ALLOW_SUPERUSER=1
      - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
      - composer self-update
      - composer install
      - find src/ -type f -name '*.php' -print0 | xargs -0 -n1 -P4 php -l -n | (! grep -v "No syntax errors detected" )
      - composer run phpcs
      - composer run phpstan
      - ./vendor/bin/phpunit -c dev/tests/project/unit.xml --testsuite Project_Unit_Tests
      - composer run yireo
      - COMPOSER_AUDIT_ABANDONED=report composer audit

A few notes on this configuration:

• Composer scripts: PHP_CodeSniffer, PHPStan and the module dependency check are defined as scripts in the project’s composer.json. This way, the pipeline and every developer run exactly the same command with exactly the same configuration.
• Module dependency check: composer run yireo runs the Yireo ExtensionChecker. It reports classes a module uses without declaring the dependency in its composer.json and module.xml, which otherwise only blows up once the module is installed somewhere else.
• Security audit: composer audit fails if an installed package has a known security advisory. With COMPOSER_AUDIT_ABANDONED=report, abandoned packages are only listed, but do not fail the build – otherwise a single abandoned dependency deep in the tree would block every pull request.
• Client-specific tests only: we only run the unit and integration tests of our own modules, not the ones from the Magento core.

Step 2: Integration Tests

This step runs the integration tests of our own modules against a real database and search engine. The setup is the same as in step 1: system packages, PHP extensions, memory limit, Composer and composer install. Then we create an empty database, point Magento’s integration test configuration to it and run PHPUnit.

The database and the search engine are defined as services at the end of the file:

definitions:
  services:
    mysql:
      image: mysql:8.4
      variables:
        MYSQL_ROOT_PASSWORD: password
    elasticsearch:
      image: lionslair/elasticsearch-bitbucket-pipelines
      memory: 2048

Bitbucket does not support multiple databases out of the box. The workaround is to define a root password and create the databases yourself via CLI. Afterwards, we adjust Magento’s integration test configuration with sed:

- step:
    name: Integration Tests
    caches:
      - composer
    services:
      - mysql
      - elasticsearch
    script:
      - apt-get update && apt-get install -y git libxslt1-dev libxml2-dev libicu-dev libpng-dev unzip openssh-client libfreetype6-dev libjpeg-dev default-mysql-client libzip-dev
      - docker-php-ext-configure gd --with-freetype --with-jpeg
      - docker-php-ext-install -j$(nproc) bcmath ftp gd intl pdo_mysql soap sockets xsl zip
      - echo 'memory_limit = 3G' >> /usr/local/etc/php/conf.d/docker-php-memlimit.ini
      - export COMPOSER_ALLOW_SUPERUSER=1
      - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
      - composer self-update
      - composer install
      - mysql -h 127.0.0.1 -u root -ppassword -e 'CREATE DATABASE `integration-tests-db`;'
      - cp dev/tests/integration/etc/install-config-mysql.php.dist dev/tests/integration/etc/install-config-mysql.php
      - sed -i -e "s/'localhost'/'127.0.0.1'/" -e "s/'123123q'/'password'/" -e "s/'magento_integration_tests'/'integration-tests-db'/" -e "/'amqp-/d" dev/tests/integration/etc/install-config-mysql.php
      - ./vendor/bin/phpunit -c $BITBUCKET_CLONE_DIR/dev/tests/project/integration.xml --testsuite Project_Integration_Tests

• Services are reachable via 127.0.0.1, not localhost – with localhost, the MySQL client tries to connect via a socket, which does not exist in the build container.
• We use Elasticsearch via the lionslair/elasticsearch-bitbucket-pipelines image instead of OpenSearch. OpenSearch requires two nodes by default, and the config option to run it as a single node cannot be passed in Bitbucket Pipelines.
• The Elasticsearch service needs more memory than the default of 1 GB, hence memory: 2048.

Step 3: End-to-End Tests Against a Real Client Database

For end-to-end tests, we use Cypress. In newer projects, we use Playwright instead – both work fine in this setup. For this step, we spin up a complete Magento shop inside the pipeline. After the usual setup from step 1, we import a stripped copy of the client database and install Magento on top of it. Then we build the frontend like on the live system, let Apache serve the shop and redirect all mails to a mail catcher. Finally, we check that no response header is too long for Apache (see below) and run the Cypress tests.

The services:

definitions:
  services:
    mysql:
      image: mysql:8.4
      variables:
        MYSQL_ROOT_PASSWORD: password
    cypress:
      image: cypress/browsers:latest
    mailhog:
      image: mailhog/mailhog
      logging:
        driver: 'none'
      ports:
        - 1025:1025 # smtp server
        - 8025:8025 # web ui

And the step itself (shortened to the interesting parts):

- step:
    name: Cypress tests
    size: 2x
    caches:
      - composer
      - node
    services:
      - mysql
      - cypress
      - mailhog
    script:
      # ... install system packages, PHP extensions and Composer as above
      - mysql -h 127.0.0.1 -u root -ppassword -e 'CREATE DATABASE `cypress-tests-db`;'
      - mkdir -p ~/.ssh && printf '%s\n' '[your.server]:22 ssh-ed25519 AAAA...' > ~/.ssh/known_hosts
      - scp user@your.server:/path/to/db.sql.gz .
      - gunzip db.sql.gz
      - mysql -h 127.0.0.1 -u root -ppassword cypress-tests-db < db.sql
      # a stripped dump leaves the authorization tables empty, see https://netz98.github.io/n98-magerun2/command-docs/db/db-add-default-authorization-entries/
      - mysql -h 127.0.0.1 -u root -ppassword cypress-tests-db -e "INSERT INTO authorization_role (role_id, parent_id, tree_level, sort_order, role_type, user_id, user_type, role_name) VALUES (1, 0, 1, 1, 'G', 0, '2', 'Administrators')"
      - mysql -h 127.0.0.1 -u root -ppassword cypress-tests-db -e "INSERT INTO authorization_rule (rule_id, role_id, resource_id, privileges, permission) VALUES (1, 1, 'Magento_Backend::all', null, 'allow')"
      - bin/magento setup:install --base-url=http://127.0.0.1/ --db-host=127.0.0.1 --db-name=cypress-tests-db --db-user=root --db-password=password --admin-user=admin --admin-password=password123! --admin-email=admin@example.com --admin-firstname=Admin --admin-lastname=Admin --no-interaction
      - bin/magento config:set web/unsecure/base_url 'http://127.0.0.1/'
      - bin/magento config:set web/secure/base_url 'http://127.0.0.1/'
      # build the frontend and deploy the static content
      - curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
      - apt-get install -y nodejs
      - npm ci
      - npm run build
      - bin/magento setup:static-content:deploy de_DE -f
      - bin/magento indexer:set-mode realtime
      # let Apache serve the cloned repository
      - sed -ri -e "s!/var/www/html!${BITBUCKET_CLONE_DIR}!g" /etc/apache2/sites-available/*.conf
      - sed -ri -e "s!/var/www/!${BITBUCKET_CLONE_DIR}!g" /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf
      - a2enmod rewrite
      - service apache2 restart
      # send all mails to MailHog
      - bin/magento config:set customsmtp/email_config/server_address 127.0.0.1
      - bin/magento config:set customsmtp/email_config/server_port 1025
      - chown -R :www-data var generated pub/static pub/media
      - chmod -R g+rwX var generated pub/static pub/media
      - >
        curl -L -s -D - http://127.0.0.1/ -o /dev/null |
        awk 'NR > 1 {if (length($0) > 8175) {print "Error: Header exceeds 8175 characters -", length($0), $0; exceeded = 1}} END {if (!exceeded) print "The header size is ok"; else exit 1}'
      - CYPRESS_INCLUDE_TAGS=dev npx cypress run
    artifacts:
      - cypress/screenshots/**
      - cypress/videos/**
      - var/log/**
      - var/report/**

A few notes on this configuration:

• size: 2x: Magento, MySQL, Cypress and the frontend build do not fit into the default 4 GB. The doubled size costs twice the build minutes.
• A stripped client database: A cronjob creates a stripped dump of the live database every night (no customers, no orders, no logs) and puts it on a server. The pipeline fetches it via scp. Testing against real products, categories and configuration finds far more issues than testing against sample data.
• Pin the SSH host key: Instead of disabling host key checking, we write the server’s host key into ~/.ssh/known_hosts. The private key itself is configured in the repository settings under Pipelines → SSH keys.
• Empty authorization tables: Stripped dumps come without the admin roles, which blocks the creation of admin users – including the one setup:install creates. The two INSERT statements recreate the administrator role, as described in the n98-magerun2 documentation.
• setup:install on an existing database: This writes a fresh app/etc/env.php and runs all pending schema and data patches of the current branch on top of the imported dump – just like a deployment would.
• Production-like frontend: We build the frontend exactly like the deployment of the respective project does. Depending on the project, that means building the theme, deploying the static content and bundling the JavaScript, e.g. with Magepack. Otherwise, we would test a frontend that never goes live.
• Apache serves the repository: The php:*-apache image serves /var/www/html. Instead of copying the whole project there, we point Apache to $BITBUCKET_CLONE_DIR.
• Mails to MailHog: All mails go to the MailHog service, so that Cypress can check them via the MailHog API – e.g. whether the order confirmation was sent.
• Header size check: Magento’s Content-Security-Policy header can become very large. By default, Apache limits a single header line to 8190 bytes (LimitRequestFieldSize). If it proxies a response with a longer header, it returns a 502 error instead of the page. That is why the pipeline fails as soon as a header exceeds 8175 characters – the limit minus a small safety margin – so we notice the problem before our customers do.
• Tags: We tag our Cypress tests with dev and live via the cypress-tags plugin. All tests run in the pipeline; the ones tagged live are safe to also run against the live shop (see below).
• Artifacts: Screenshots, videos and the Magento logs are stored as artifacts. If a test fails, you can download them from the pipeline result. That is the fastest way to find out why something failed.

Deployments

Deployments are custom pipelines, which we trigger manually for a branch. The first step sets up PHP and Composer like the test steps, installs the project dependencies – including Deployer – and deploys the branch. After a deployment to the live system, a second step runs a few Cypress smoke tests against the live shop:

custom:
  Deploy to www.example.com:
    - step:
        name: Deploy to www.example.com
        caches:
          - composer
        image: php:8.4
        deployment: www_example_com
        script:
          - apt-get update && apt-get install -y git libxslt1-dev libxml2-dev libicu-dev libpng-dev libzip-dev unzip openssh-client rsync
          - docker-php-ext-install -j$(nproc) bcmath ftp gd intl pdo_mysql soap sockets xsl zip
          - export COMPOSER_ALLOW_SUPERUSER=1
          - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
          - composer self-update
          - composer install
          - chmod 777 /tmp/
          - ./vendor/bin/dep deploy --branch ${BITBUCKET_BRANCH} www_example_com -vvv
    - step:
        name: Live Cypress smoke test
        caches:
          - node
        services:
          - cypress
        image: cypress/browsers:latest
        script:
          - npm ci
          - CYPRESS_MAGENTO2_BASE_URL=https://www.example.com CYPRESS_INCLUDE_TAGS=live npx cypress run
        artifacts:
          - cypress/screenshots/**
          - cypress/videos/**

A few notes on this configuration:

• deployment: Using Bitbucket deployment environments gives you a history of what was deployed where and when, and lets you restrict who may deploy to production.
• Deployer as a Composer dependency: Deployer is required in the project’s composer.json, so its version is pinned in composer.lock like any other package.
• openssh-client and rsync: Deployer connects to the server via SSH and uploads files via rsync, so both need to be installed. The SSH key is configured in the repository settings under Pipelines → SSH keys.
• Live smoke test: Directly after a production deployment, the live tagged Cypress tests run against the live shop. If something essential is broken, we know it within minutes – and not when the first customer complains.

Getting Notified When a Pipeline Fails

A failing pipeline is useless if nobody notices. Every step has an after-script, which runs no matter if the step failed or succeeded:

after-script:
  - if [ "$BITBUCKET_EXIT_CODE" -ne 0 ]; then curl -X GET -u "$WEBHOOK_USER:$WEBHOOK_PASSWORD" "https://your-n8n-instance/webhook/.../bitbucket-pipeline-failed/branch/$BITBUCKET_BRANCH/user/$BITBUCKET_STEP_TRIGGERER_UUID/workspace/$BITBUCKET_WORKSPACE/repo/$BITBUCKET_REPO_SLUG/build/$BITBUCKET_BUILD_NUMBER"; fi

If the step failed, it calls an n8n webhook with the branch, the person who triggered the build, the repository and the build number. n8n then posts a message with a link to the failed build to the project’s Slack channel. Keep the credentials in secured repository variables instead of the YAML file.

Debugging Bitbucket Pipelines

Bitbucket Pipelines is nice, but a hell to debug. What helps us:

• Artifacts: Store everything you might need to understand a failure: Cypress screenshots and videos, var/log, var/report and even the system’s /var/log.
• after-script: It runs even if the step failed, so it is the right place to print logs or collect test results.
• Reproduce locally: Run the same Docker image locally (docker run -it -v $(pwd):/app php:8.4-apache bash) and execute the script line by line. That is much faster than pushing a commit for every attempt.
• Composer scripts: As the checks are Composer scripts, a failing check can be reproduced locally with exactly the same command.

General Tips

Finally, some things which apply to every step and are easy to miss:

• Enable the composer cache (and the node cache, if you build a frontend). It speeds up every build and saves build minutes.
• Pass -j$(nproc) to docker-php-ext-install, so that the extensions are compiled in parallel.
• Start from a slim official image like php:8.4-apache and install the required dependencies yourself. That way, the image contains only what you actually need.
• Raise the memory limit to 3G. Magento needs a lot of memory for composer install, setup:di:compile and PHPStan.
• Set COMPOSER_ALLOW_SUPERUSER=1. The pipeline runs as root, and without this variable Composer disables plugins – including dealerdirect/phpcodesniffer-composer-installer, which registers the coding standards for PHP_CodeSniffer.

It took us quite a few evenings to get all of this to work, so we hope that we can save you some time with this post 🙂 If you have any questions, feel free to contact us!