• RegressionTest

CAcert Regression Test

We are currently building up a regression test system, to automatically test the functionality of the CAcert website.

Regression tests against the production system must be authorized by at least 2 people from the core-team, and monitored by at least one core-team member. Regression code that runs against the production system must not be commited to the SVN, or otherwise publizised

There is a new regression testing system that we should try out: Selenium

Where to start?

You can get our current regression tests from a subversion repostory at: http://svn.cacert.org/CAcert/Regression/

A live site running the tests software is available at http://test.chost.de/Regression/ - username and password is both cacert.

The tests are currently not testing the main CAcert website but https://www.test1.cacert.at/

Several tests have been written so far. The testing framework we use is SimpleTest. Basically a test is a script simulating the user/admin/etc input to CAcert web site and checking the results.

Please protect your test installation and framework from indexing by a search engine. You can do this by using a "robots.txt" in your web directory.

Tests Already Implemented

general-website.php

• front page available via http/https?
• contact page available via http/https?
• Root Cert fingerprints correct via http/https?
• first line of Root Cert (PEM) correct via http/https?
• (Changing language to de_DE possibe via http/https?) not testable testserver
• is wiki.cacert.org online?
• is blog.cacert.org online?
• is bugs.cacert.org online?

login.php

• logon with empty password possible?
• logon with empty email address possible?
• logon with wrong password possible?
• normal logon possible?
• logout possible?

lostpassword.php

• lost password page available?
• empty email and date of birth catched?
• empty date of birth catched?
• lost password step 2 available?
• empty lost password answers catched?
• correct answers but no new password catched?
• setting new password using lost password questions possible?

join.php

• empty password detected?
• missing lost password questions/answers detected?
• missing date of birth detected?
• missing email detected?
• missing last 2 lost password questions/answers detected?
• simple passwords detected?
• different passwords detected?
• using already registered emails addresses possible?
• joining possible?
• login possible? (link in verification email is automatically followed)

newemail.php

• login possible?
• adding new email possible?
• does verification of new emails work? (link in verification email is automatically followed)
• deletion of email addresses possible?

change-details-password.php

• login possible?
• setting user details do test values possible?
• changing password possible?
• bad passwords (short, simple) catched?

additional tests

• assuring a user
• revoking a user assurance by admin

Planned Tests

The following scenarios should be covered by the Regression Test system. If you start working on a scenario, please add your name besides the scenario on this page, to make sure, that somebody else isn't doing the same thing at the same time.

Please see UseCases

• adding DB access to compare/simulate user behaviour (mail probes...) - bluec says that this should be handles using small scripts behind the test email addresses. Its already working more or less.
• Creation of the same user
• creation of a "random named" user
• deletion of the previous user by an admin.
• login of an unknown user
• adding a domain for a user
• creation of a certificate with a properly crafted CSR
• creation of a certificate with a domain name not validated in the user account
• ...
• login with a wrong password
• login with a password of another account
• creation of a user with a very long name
• creation of a user with special UTF-8 characters
• creation of a user with invalid UTF-8 characters
• issuing certificates with UTF-8 characters
• issuing wildcard certificates
• issuing wildcard certificates for subdomains
• issuing wildcard certificates with www.*.domain.com
• issuing too many points
• issuing certificates with CSRs with broken Signature
• issuing a class1 certificate
• issuing a class3 certificate
• issuing a certificate, modifying the server time into the future, past the certificate expiry time, verifying the certificate with OCSP
• issuing a certificate, modifying the server time into the past, verify the certificate with OCSP
• ...

Organisation account test

• creating a new organisation (admin) => DN, domain names, organisation admins

• creating an email certificate and testing DN values
• creating a server certificate and testing DN values

History

Requirements

At the beginning, we started collecting the requirements we have for a CAcert regression-test framework:

Requirements:

• Remote testing
• HTTPS
• Login
• Cookies
• Transaction oriented

Frameworks evaluated

Then we evaluated the following software packages: