The case to use PCOV to generate test coverage results

• 2 min read

Test coverage is a metric which is often overvalued. Developers often assume they have to get to 100% coverage in their applications at all costs.
I think this is the wrong mindset. Not every single line of your application has to be tested. Writing Unit tests to test that a method getEmail() actually returns an email address is in my opinion, wasted time. (For packages, I agree that a high test coverage can be benefitial. It's usually also easier to do.)

However, knowing how much of your application is covered by tests can be a good indicator in which shape your project is. At work™, we recently started generating code coverage for all our projects.

In a particular large project (+1500 tests) the addition of running the test suite with xdebug enabled slowed the CI run down quite a bit. Normally the test suite takes 2 to 3 minutes to run. With xdebug enabled it took ~30 minutes. 😱

We're using GitHub Actions to run the test suite. In particular, we're using the shivammathur/setup-php-Action to create the correct PHP environment.
While figuring out, how we could speed up the CI run, I've stumbled upon the PCOV-option of the Action. PCOV is a driver built for generating code coverage. One of its main claims is, that it's much faster than xdebug. (Note: There will be a differerence in the coverage reporting between PCOV and xdebug (see here)).

Is it faster? Let's try it out.
These are the changes I had to make to our workflow. Instead of using coverage: xdebug we're now using coverage: pcov. In addition, I had to set ini-values: pcov.directory=app. This tells the driver where our application code lives.

name: Setup PHP
uses: shivammathur/setup-php@v2
with:
  php-version: 7.4
  extension: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
 coverage: xdebug
 coverage: pcov
 ini-values: pcov.directory=app

And this is the entire GitHub Actions workflow we're using right now.

name: Tests (with Coverage)

on: push

jobs:
    tests:
        runs-on: ubuntu-latest

        steps:
            - name: Checkout Code
              uses: actions/checkout@v2

            - name: Setup PHP
              uses: shivammathur/setup-php@v2
              with:
                  php-version: 7.4
                  extension: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
                  ini-values: pcov.directory=app
                  coverage: pcov

            - name: Install Composer Dependencies
              run: composer install --prefer-dist --no-interaction --no-suggest

            - name: Copy Environment File
              run: cp .env.example .env

            - name: Generate Application Key
              run: php artisan key:generate

            - name: Execute PHPUnit Tests
              run: vendor/bin/phpunit --coverage-text --coverage-clover=coverage.xml

We've pushed the change in the mentioned work project and the time for running the test suite dropped significantly. From ~30 minutes to the ususal 2 to 3 minutes!

I was so happy to report this to the rest of the team. And I think it's also amazing, how low effort the change was. Actions like setup-php makes working with and tweaking Continous Integration workflows on GitHub so much easier.