The case to use PCOV to generate test coverage results
Test coverage is a metric which is often overvalued. Developers often assume they have to get to 100% coverage in their applications at all costs.
I think this is the wrong mindset. Not every single line of your application has to be tested. Writing Unit tests to test that a method
getEmail() actually returns an email address is in my opinion, wasted time. (For packages, I agree that a high test coverage can be benefitial. It's usually also easier to do.)
However, knowing how much of your application is covered by tests can be a good indicator in which shape your project is. At work™, we recently started generating code coverage for all our projects.
In a particular large project (+1500 tests) the addition of running the test suite with xdebug enabled slowed the CI run down quite a bit. Normally the test suite takes 2 to 3 minutes to run. With xdebug enabled it took ~30 minutes. 😱
We're using GitHub Actions to run the test suite. In particular, we're using the shivammathur/setup-php-Action to create the correct PHP environment.
While figuring out, how we could speed up the CI run, I've stumbled upon the PCOV-option of the Action. PCOV is a driver built for generating code coverage. One of its main claims is, that it's much faster than xdebug. (Note: There will be a differerence in the coverage reporting between PCOV and xdebug (see here)).
Is it faster? Let's try it out.
These are the changes I had to make to our workflow. Instead of using
coverage: xdebug we're now using
In addition, I had to set
ini-values: pcov.directory=app. This tells the driver where our application code lives.
- name: Setup PHP uses: shivammathur/setup-php@v2 with: php-version: 7.4 extension: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick - coverage: xdebug + coverage: pcov + ini-values: pcov.directory=app
And this is the entire GitHub Actions workflow we're using right now.
name: Tests (with Coverage) on: push jobs: tests: runs-on: ubuntu-latest steps: - name: Checkout Code uses: actions/checkout@v2 - name: Setup PHP uses: shivammathur/setup-php@v2 with: php-version: 7.4 extension: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick ini-values: pcov.directory=app coverage: pcov - name: Install Composer Dependencies run: composer install --prefer-dist --no-interaction --no-suggest - name: Copy Environment File run: cp .env.example .env - name: Generate Application Key run: php artisan key:generate - name: Execute PHPUnit Tests run: vendor/bin/phpunit --coverage-text --coverage-clover=coverage.xml
We've pushed the change in the mentioned work project and the time for running the test suite dropped significantly. From ~30 minutes to the ususal 2 to 3 minutes!
I was so happy to report this to the rest of the team. And I think it's also amazing, how low effort the change was. Actions like setup-php makes working with and tweaking Continous Integration workflows on GitHub so much easier.