HN Explorer

Ask HN: How do you test CMS-based websites?

4 points by andrei_says_ ↗ HN
Running a large complex content website with frequent code deployments and content updates. We do have manual testing in place, but I would like to add some automated testing.

Things I'll want to test:

* pages loading properly without errors * module-based content shows (as in excerpt of latest blog posts show on the home page etc.) * page modules like accordions and carousels function properly * navigation functions properly * forms submit (might have to forego this one as most forms depend on recaptcha submission)

What I have looked at:

Selenium -- the firefox ide seems a bit buggy

Katalon studio -- the files it produces are not fully compatible with Selenium (last time I checked).

We do have Ruby experience so also considering watir although not much experience with it. IDE-based test creation would be much faster though.

What do you use for website testing? I am interested not only in tooling , but also process: how do you create the tests, what do you test, how often, how do you audit the results?

How do you maintain the tests (content changes frequently).

Thank you.

8 comments

[ 3.7 ms ] story [ 36.7 ms ] thread
dylz
Headless Chrome running in lambdas/cloud functions. CI/CD validates this with each push.

In the dev/test/staging environments you can continually do things like post 3 new articles with random text, then ensure that the latest blog posts show those 3 in the correct order, etc.

andrei_says_
Could you provide a bit more detail? How do you script the test? How do you maintain it against content changes?

And, how do you apply it against production site which may display defects different environments?

The CMS we use does not have any automation so automating posting and analyzing results will be out of scope.

wprapido
If Selenium doesn't work, give Puppeteer a go. It's NodeJS-based though.

How do you go about? Pretty simple. Define criteria, run Chrome Headless against it, and check results. Jest is a nice add on in your toolchain!

As there are many use cases and a million criteria to test agains, coming up with a one size fits all solution is difficult. Katalon is perhaps closest to it, but it has many shortcomings.

Some food for thought. Rely more on screenshots and compare them with what you expect. I'm writing a simple downtime checker as we speak and one of the key features is very basic analytics that compares screenshot to blank screens, generic error pages, parked domain landing pages, etc.

andrei_says_
What are some of Katalon's shortcomings you've experienced? Asking because it's one of the top options right now.
wprapido
Doesn't play nice with other tools
dyeje
Never done it before, but I would try using selenium with rspec or minitest. I would use webdriver to do an actual browser as well, if your site has any javascript.
ryancp
I am working on an automated way to test what you are describing. It's a codeless functional testing SaaS for browser based apps/sites. It's still new and I am looking for early customers to gather feedback and validate the product. Check it out at https://ninjaqaiden.com and I'd love to help see if it fits your needs.
pryelluw
Id love to help more in depth. Would you mind shooting me an email? pryelluw at gmail . com

Not trying to sell you or anything. Just that this is not something I can briefly answer on HN.