Perform Percy snapshotting in end to end tests

[gumaerc]

Description/Context

Browserstack Percy is a product that takes snapshots of websites and performs automatic visual comparison, generating images that show the differences between two snapshots. We should add this to our end to end tests so visual comparisons can be made both from branch to branch and also from commit to commit if necessary.

Plan/Design

• Add the @percy/cli and @percy/playwright to dev dependencies
• Add Percy snapshots to the appropriate tests:
  • The main ocw-www home page
  • The search page
  • The course home page
  • A course content page
• Make sure the proper env variables are set up in RC / production before merging PR's related to this

[gumaerc]

Here's some code that I wrote to do some experimental testing with Percy. This just takes a snapshot of the course home page:

import playwright from "playwright"
import { env } from "../../env"
import { test, expect } from "@playwright/test"
import { getFirstAfter, CoursePage } from "../util"
const percySnapshot = require('@percy/playwright')

const browsers = {
  "firefox": playwright.firefox,
  "chromium": playwright.chromium
}

test("Loads the course home page", async ({ browserName }) => {
  const browser = await browsers[browserName].launch()
  const page = await browser.newPage()
  const course = new CoursePage(page, "course")
  await course.goto()
  await percySnapshot(page, `Loads the course home page in ${browserName}`)
  await browser.close()
})

One peculiarity that I found while experimenting with this is that you need to run the browser.close() function in order for Percy to consider the session "finished" and take the snapshot, otherwise it will wait indefinitely. All of our tests only use the page fixture, which doesn't include the browser object. If I used the browser fixture and run browser.close() on that, I would get an error from the next test that "the browser has been closed" even though each Playwright test is supposed to spin up its own browser window and close it at the end. So, I found that the best way to do this if you want a Percy snapshot in the test is to explicitly create a browser in the manner above. Also, we would need to create a new package.json script that runs the e2e tests with Percy:

"test:e2e:percy": "yarn with-env --dev -- percy exec -- ./package_scripts/test_e2e.sh",

[pdpinch]

If we had added the tests as described here, what would have happened in a case like #1379?

[HussainTaj-arbisoft]

One peculiarity that I found while experimenting with this is that you need to run the browser.close() function in order for Percy to consider the session "finished" and take the snapshot, otherwise it will wait indefinitely.

I wonder whether or not it works if we use page.close instead of browser.close.

[gumaerc]

If we had added the tests as described here, what would have happened in a case like #1379?

I'm not sure we would have caught this issue, because it requires a pretty specific scenario where the course description uses a shortcode and is also longer than 320 characters. If we had the testing in place before the issue happened, and added a playwright test to check truncation beforehand it's possible we would have noticed it. It's likely we would add a Percy snapshot to the course home page, so if there was a difference visually between the unexpanded descriptions between two branches then it would have been flagged. It's also possible that a Playwright test checking for unrendered markdown in the description could have seen this.