Testing

Testing is extremely important for evaluating the overall health, durability, safety, security and adherence to standards of a piece of software or system.

Criteria

7 Testing Criteria:

  1. Beta Testing (does it work?)
  2. Usability Testing (is it simple/usable?)
  3. Content Review (who can use it? practicality of material...)
  4. Learning Outcomes (how does it get used/absorbed?)
  5. Integration into Daily Life (when/where does it fit in existing systems/usages? i.e. training/course curriculums)
  6. Impact on Daily Life (what effect does it have? i.e. medical clinic practice, scheduling appointments, send msgs)
  7. Validity Testing (why do this task or add this feature? are the results relevant/valid...)

Test Strategies

Phased

The "traditional" approach to Testing is that it proceeds after (and in Waterfall strictly only after) all software, applications, supporting systems, etc have been delivered and fully integrated. At the end of the project, the time between "software works" and "software delivered to customer" is often dubbed the "Testing Phase". While this ensures that only "one major pass" of testing theoretically needs to get done, with much lower needs for frequent minor regression tests, it is often too naive to satisfy real-world (constantly changing) business needs.

Often in Agile Development environment, the Phased approach must be at the very least modified to have several "medium sized" iterative phases. Other strategies are to give up on a Phased Testing approach entirely and begin Continuous Testing.

Continuous Testing

• wikipedia: Continuous testing
• Getting started with "Headless Chrome" browser: https://developers.google.com/web/updates/2017/04/headless-chrome
• Using headless Chrome as an automated screenshot tool: https://medium.com/@dschnr/using-headless-chrome-as-an-automated-screenshot-tool-4b07dffba79a

Shift Left

• wikipedia: Shift left testing
• What is Shift-left? SL Testing explained: http://www.bmc.com/blogs/what-is-shift-left-shift-left-testing-explained/
• Shift-left testing and overall quality: http://www.bitwiseglobal.com/blogs/shift-left-testing-and-overall-quality/
• What Shift-left testing means: https://smartbear.com/learn/automated-testing/shifting-left-in-testing/
• Shift-left at IBM: https://www.slideshare.net/Urbancode/shift-left
• Four types of "Shift-left" Testing: https://insights.sei.cmu.edu/sei_blog/2015/03/four-types-of-shift-left-testing.html
• Shift-left Testing not a new concept (2001 article): http://www.drdobbs.com/shift-left-testing/184404768
• Shift-left in Performance Testing: https://www.infoq.com/articles/shift-left-performance-testing
• Shift-left with Test Automation in Agile Dev environments: http://www.bcs.org/upload/pdf/shift-left-test-automation-in-agile-environments.pdf
• Shift-left - why you need to perform testing earlier in Dev cycle: https://dzone.com/articles/shift-left-approach-why-do-you-need-to-perform-ear
• We're all Testers now -- getting the organization ready for Shift-left automation: https://techbeacon.com/were-all-testers-now-5-steps-get-ready-shift-left-automation

Shift Right

• 8 Reasons to Shift Testing Right: https://www.linkedin.com/pulse/8-reasons-shift-testing-right-lanette-creamer/
• Shift-right -- Test microservices in the wild to tame DevOps: https://techbeacon.com/shift-right-test-microservices-wild-tame-devops

Types of Testing

• wikipedia: Software testing

Unit

Unit Testing (sometimes referred to as "local testing" or abbreviated UT) is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use. Intuitively, one can view a unit as the smallest testable part of an application. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method. Unit tests are short code fragments created by programmers or occasionally by white box testers during the development process. Ideally, each test case is independent from the others. Substitutes such as method stubs, mock objects, fakes, and test harnesses can be used to assist testing a module in isolation. Unit tests are typically written and run by software developers to ensure that code meets its design and behaves as intended.

• wikipedia: Unit testing
• See also: jUnit

CBUT

Component-Based Usability Testing (CBUT) is a testing approach which aims at empirically testing the usability of an interaction component. The latter is defined as an elementary unit of an interactive system, on which behaviour-based evaluation is possible. For this, a component needs to have an independent, and by the user perceivable and controllable state, such as a radio button, a slider or a whole word processor application. The CBUT approach can be regarded as part of component-based software engineering branch of software engineering.

• wikipedia: Usability testing
• wikipedia: Component-based usability testing

Functional

Functional Testing is a form of test upon an entire application in isolation from any external dependencies. In this sense the focus is the same as a Unit Test but the context is the entire application (minus any external dependencies) rather than its individual parts/units.

API Testing

A key form of Functional Testing is API Testing which hits an API directly without use of the application's User Interface (UI) which often invokes all phases and dependencies at once (which is more of a holistic/system-level test). Calling individual backend APIs that are depended upon by the front-end can help for monitoring the health of the application and act as a trigger as to the readiness for moving to the next stage of testing for running UI-centric system tests and/or integration. In addition, failures in Functional API Tests which are configured to run on a regular basis can be hooked into notifications to support teams to avoid time wasted on Root Cause Analysis when an area of the application fails due to a failure in a depended upon API.

• SoapUI: http://soapui.org
• Swagger Inspector -- Test & auto-generate OpenAPI documentation for any API: https://inspector.swagger.io/builder | DOCS
• Postman: https://www.getpostman.com/

Integration

Integration Testing (sometimes called integration and testing, abbreviated I&T) is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before validation testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.

• wikipedia: Integration testing

Smoke

Smoke Testing for software is typically trying the major functions of the whole piece of software (or recently updated component) before carrying out formal testing, and comes from Smoke Testing in electrical engineering where the term literally means "looking for smoke" when powering electrical items or new connections within/between them for the first time. This type of testing is usually conducted immediately after the integration of software or components into larger systems, as such it could be considered as and conducted within either a part of Integration Testing or System Testing.

• wikipedia: Smoke testing (software)

Automated

Automated Tests are repeatable, measurable and isolated test cases that can be run in unison with many other test cases with few or no dependencies. Software Tests have to be repeated often during development cycles to ensure quality. Every time source code is modified software tests should be repeated. For each release of the software it may be tested on all supported operating systems and hardware configurations. Manually repeating these tests is costly and time consuming. Once created, automated tests can be run over and over again at no additional cost and they are much faster than manual tests. Automated software testing can reduce the time to run repetitive tests from days to hours.

• wikipedia: Test automation
• THE Internet app: https://the-internet.herokuapp.com/ (comprehensive "generic" Webpage/WebApp functionality for Automated Testing "training/practice")</ref>

System

System Testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. As a rule, system testing takes, as its input, all of the "integrated" software components that have passed integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of integration testing is to detect any inconsistencies between the software units that are integrated together (called assemblages) or between any of the assemblages and the hardware. System testing is a more limited type of testing; it seeks to detect defects both within the "inter-assemblages" and also within the system as a whole.

• wikipedia: System testing

Regression

Regression Testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes such as enhancements, patches or configuration changes, have been made to them. The intent of regression testing is to ensure that changes such as those mentioned above have not introduced new faults. One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software. Common methods of regression testing include rerunning previously completed tests and checking whether program behavior has changed and whether previously fixed faults have re-emerged. Regression testing can be performed to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change.

• wikipedia: Regression testing

Acceptance

Tests conducted to determine if the requirements of a specification or contract are met.

• wikipedia: Acceptance_testing

UAT

User Acceptance Testing (UAT) is the process of verifying that a solution works for the user. It is not system testing (ensuring software does not crash and meets documented requirements), but rather is there to ensure that the solution will work for the user i.e. test the user accepts the solution (software vendors often refer to as Beta testing). This testing should be undertaken by a Subject-Matter Expert (SME), preferably the owner or client of the solution under test, and provide a summary of the findings for confirmation to proceed after trial or review. In software development, UAT as one of the final stages of a project often occurs before a client or customer accepts the new system. Users of the system perform tests in line with what would occur in real life scenarios. The UAT acts as a final verification of the required business functionality and proper functioning of the system, emulating real-world usage conditions on behalf of the paying client or a specific large customer. If the software works as required and without issues during normal use, one can reasonably extrapolate the same level of stability in production. User tests, usually performed by clients or by end-users, do not normally focus on identifying simple problems such as spelling errors and cosmetic problems, nor showstopper defects, such as software crashes; testers and developers previously identify and fix these issues during earlier unit testing, integration testing, and system testing phases.

• wikipedia: Acceptance_testing#User_acceptance_testing (UAT)

A/B

In Marketing and Business Intelligence, A/B testing is jargon for a randomized experiment with two variants, A and B, which are the control and treatment in the controlled experiment. It is a form of statistical hypothesis testing with two variants leading to the technical term, Two-sample hypothesis testing, used in the field of statistics. Other terms used for this method include bucket tests and split testing but these terms have a wider applicability to more than two variants. In online settings, such as web design (especially user experience design), the goal is to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). Formally the current web page is associated with the null hypothesis. As the name implies, two versions (A and B) are compared, which are identical except for one variation that might affect a user's behavior. Version A might be the currently used version (control), while Version B is modified in some respect (treatment). For instance, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales.

• wikipedia: A/B testing
• Optimizely: https://www.optimizely.com/ | A/B Testing
• Google Optimize: https://www.google.com/analytics/optimize/
• Web Analytics Automation Testing Framework (WAAT): https://essenceoftesting.blogspot.ca/2017/10/analytics-forgotten-child.html | SRC
• “Test What Matters: Automated Analytics Testing” – by Marcus Merrell

(WEBINAR): https://applitools.com/blog/webinar-recording-test-what-matters-automated-analytics-testing

Mobile

• wikipedia: Mobile application testing
• Appium: http://appium.io/ | DOCS | SRC
• What is Mobile Testing?: https://smartbear.com/learn/software-testing/what-is-mobile-testing/
• The Essential Guideto Mobile App Testing: http://core.ecu.edu/STRG/materials/uTest_eBook_Mobile_Testing.pdf
• Mobile Testing Tutorial: https://www.tutorialspoint.com/mobile_testing/index.htm
• Introducing MAD LAB - for Mobile Automation: https://essenceoftesting.blogspot.ca/2017/04/introducing-mad-lab-for-mobile.html
• The iOS Testing Manifesto: https://medium.com/@hesham.salman/the-ios-testing-manifesto-e1bc821cc4c3

Layout

Layout Testing is a form of testing focused on ensuring that responsive web designs retain the desired ratios and relative distances between page elements as resizing occurs and/or breakpoints are toggled.

• Galen Framework: http://galenframework.com (Galen is the leading OSS Layout Testing framework)

Performance

Performance Testing in software involves ensuring the quality and reliability of a particular set of that software as well as its components, and the inter-connections between them, in order to get an accurate picture of the health of the whole system. It includes a number of techniques including: Load Testing, Disaster Recovery Preparedness, High-Traffic Response Times, Failover Management, etc.

• wikipedia: Software performance testing

Load Testing

• wikipedia: Load testing

Penetration Testing

Testing for Security vulnerabilities by Hacking, DDOS'ing, etc.

• wikipedia: Penetration testing

QA

Quality Assurance (QA) is a key aspect of the testing process and should be done throughout the development process, as new components are added.

• wikipedia: Quality Assurance
• wikipedia: Root cause
• wikipedia: Root cause analysis (abbreviated RCA)
• wikipedia: Eight Disciplines Problem Solving
• wikipedia: Seven Basic Tools of Quality
• wikipedia: Quality management system
• wikipedia: Five Ws (aka. 5 questioons... Who, What, When, Where, Why... bonus question for brainstorming/problem-solving is How)
• wikipedia: 5 Whys (aka. Why? five times... each one getting closer to the root cause and sufficient information to then form solutions)

Continuous Testing

Continuous Testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. For Continuous Testing to work, the scope of testing should include validation of requirements from the bottom up, as well as (if they are used) verification of User Stories to assess the realization of system requirements associated with achieving business goals that prompted codifying of the requirements in the first place.

• wikipedia: Continuous testing
• Continuous Testing - What Exactly is It?: http://java.dzone.com/articles/continuous-testing-what

TDD

Test-Driven Development (TDD) is an approach to software development where Requirements are turned into very specific test case(s), then the software is improved/enhanced to pass the new test(s). This is in contrast to more traditional software development that allows software to be added that is not proven to meet requirements by a specific passing test case/suite.

The following are the 5 steps for proper TDD:

1. Add a test
2. Run all tests and see if the new test fails
3. Write the code
4. Run tests
5. Refactor code

• wikipedia: Test-driven_development
• Test Driven Development (TDD) - Best Practices Using Java Examples: https://technologyconversations.com/2013/12/24/test-driven-development-tdd-best-practices-using-java-examples-2/

BDD

Behavior Driven Development (BDD) combines practices from TDD and from Acceptance Test Driven Development (ATDD); like these, it includes the practice of writing tests first, but additionally focuses on tests which describe behavior, rather than emphasizing tests which validate or verify status of a unit of implementation. BDD requires clear understanding of the software's intended behavior and actual usage, through discussion with a project's stakeholders.

The practices of BDD include:

1. Establishing goals of any stakeholders directly related to the business.
2. Establishing features which will achieve those goals using feature injections.
3. Establishing behavior of the application or of units of code using examples.
4. Automating the selected examples to provide quick feedback, transparency & regression testing.

• wikipedia: Behavior-driven development
• Introducing BDD: https://dannorth.net/introducing-bdd/
• Translating TDD to BDD: https://lizkeogh.com/2009/11/06/translating-tdd-to-bdd/
• BDD in Action (BOOK): https://www.manning.com/books/bdd-in-action (Behavior-Driven Development for the whole software lifecycle, broadly considered "the only true step-by-step guide" to BDD)

Code Coverage

Code Coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.

• EclEmma - Java Code Coverage for Eclipse: http://www.eclemma.org/
• Cobertura (Eclipse plugin): http://cobertura.sourceforge.net/
• Code Coverage for NetBeans modules: http://wiki.netbeans.org/CodeCoverage
• Is Code Coverage Important?: http://java.dzone.com/articles/is-code-coverage-important

Code Review

Code Review (also referred to as peer review) is the systematic examination of computer source code upon its being submitted (i.e. committed to SCM) for consideration for inclusion in a broader code base. It is intended to find mistakes overlooked in the initial development phase, improving the overall quality of software. Reviews are done in various forms such as pair programming, informal walkthroughs, and formal inspections.

• wikipedia: Code review
• Code Review guidelines: https://docs.google.com/document/d/101FGZ0B1QqrPKl5F9x9-2vpr4xm6ku8sGqG1nALRwKA/edit
• The Code Review -- The Most Important Developer Practice: https://dzone.com/articles/the-most-important-developer-practice
• 11 Best Practices for Peer Code Review: http://support.smartbear.com/support/media/resources/cc/11_Best_Practices_for_Peer_Code_Review.pdf
• Code Review tips: https://github.com/ryanmcdermott/code-review-tips#basics

Static Code Analysis

• wikipedia: List_of_tools_for_static_code_analysis#Java
• PMD: https://pmd.github.io/ | DOCS - CLI
• FindBugs: http://findbugs.sourceforge.net | DOCS - CLI
• Checkstyle: http://checkstyle.sourceforge.net/ | DOCS - CLI
• JS lint: http://jslint.com
• JavaScript Lint: http://www.javascriptlint.com (different from the original JSlint)
• JShint: http://jshint.com
• ESlint: https://eslint.org
• CSS lint: http://csslint.net
• Stylelint: https://stylelint.io
• Jenkins plugins -- Static Code Analysis: https://wiki.jenkins.io/display/JENKINS/Static+Code+Analysis+Plug-ins (aggregation tool that depends on other Linters/Analyzers such as PMD, Findbugs, Checkstyle, OWASP Dependency Checker, etc)

Debugging

• The differences between Testing & Debugging: https://dzone.com/articles/the-differences-between-testing-and-debugging
• Debugging Tools for Windows: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools
• OllyDbgG: http://www.ollydbg.de/ (native x86 app debugger)
• IDA: https://www.hex-rays.com/products/ida/ (Windows, Linux or Mac OS X hosted multi-processor disassembler and debugger)

Tools

• BrowserStack: http://www.browserstack.com/ | DEMO (by John Resig, great for testing webapps on various browser combos)
• Appium: http://appium.io/ (Mobile App emulation/testing)
• iOS & Android Emulator on Windows: https://www.smartface.io/ios-and-android-emulator-on-windows/
• BrowserLab from Adobe: https://browserlab.adobe.com/ (VNC type remote access into a virtualized machine with many Device/OS/Browser combos)
• Browserling - Embed and Automate Real Browsers on Demand: https://www.browserling.com | API
• WebPageTest: http://www.webpagetest.org/ | EXAMPLE (page speed testing and optimization checker)
• Y-Slow?: https://addons.mozilla.org/en-US/firefox/addon/yslow/ (FF plugin to perform basic page load-time optimization checks and suggest speed enhancements)
• SLOCCount -- a set of tools for counting physical Source Lines of Code (SLOC) in a large number of languages of a potentially large set of programs: http://www.dwheeler.com/sloccount/
• Codility: http://codility.com/

Test Management

HP QualityCenter

• HP QualityCenter: https://software.microfocus.com/en-us/products/quality-center-quality-management/overview | HELP | DOCS

SpiraTest

• SpiraTeam: http://www.inflectra.com/SpiraTeam/ | DEMO | PLUGINS | DOCS | SUPPORT
• SpiraTest/SpiraTeam -- Bug Tracking Integration: https://www.inflectra.com/Documents/SpiraTestPlanTeam%20External%20Bug%20Tracking%20Integration%20Guide.pdf
• SpiraTest/SpiraTeam -- Web Services: http://api.inflectra.com/Spira/Services/ | SOAP | REST
• SpiraTest/SpiraTeam -- Automated Testing: http://www.inflectra.com/Documents/SpiraTest-Team%20Automated%20Testing%20Integration%20Guide.pdf
• https://www.inflectra.com/Documents/SpiraTestPlanTeam-Build-Server-Integration-Guide.pdf | [ ADD-ONS]

Simulators/Emulators

• Ripple Emulator - Chrome plugin: https://chrome.google.com/webstore/detail/ripple-emulator-beta/geelfhphabnejjhdalkjhgipohgpdnoc?hl=en
• Genymotion - faster Android emulator: https://www.genymotion.com/#!/

Web Services

SOAP UI

• SOAP UI: http://www.soapui.org/
• SOAP UI, Regression Testing of Human based processes, using Groovy scripts or SOAP UI Steps: http://intaliotempo.wordpress.com/2009/12/01/soap-ui-regression-testing-of-human-based-processes/
• SoapUITestCaseRunner to run SoapUI Web Service tests from JUnit: http://www.soapui.org/userguide/commandline/functional.html
• soapUI vs. JUnit: http://myarch.com/soapui-vs-junit
• Test JMS from SOAP UI: http://www.soapui.org/tutorials/jms/jms-tutorial.html
• Web Service Testing with soapUI Part 2 – A Simple Unit Test: http://qatinsights.com/?p=345
• Running soapUI Unit Tests in Hudson/Jenkins: http://blogs.sap.com/2010/12/07/running-soapui-unit-tests-in-hudson/ (Test Automation / Continuous Testing - OPTION #1 -- use [Maven SoapUI Plugin])
• Functional/headless testing via SOAP UI (for running from CLI, CI build/test server, automatically on system login, etc): https://www.soapui.org/test-automation/running-from-command-line/functional-tests.html
• Automating SoapUI test runs with Command Line Execution: https://support.smartbear.com/readyapi/docs/soapui/running/automating/cli.html

Load UI

• loadUI: http://sourceforge.net/projects/loadui/

Language-Specific

AJAX

• Jasmine: http://pivotal.github.com/jasmine/ (behavior-driven development framework for testing JavaScript code)

Selenium

• Selenium -- Web Application: http://seleniumhq.org/ | PROJECT | FF 50.x "Portable App" (Selenium is a platform consisting of language-agnostic automated test suites, plus a Firefox IDE/plugin for recording interactions as test cases then exporting to your programming language of choice: Java/C#/Python/NodeJS, other converters exist too)
• Selenium - Keyboard mappings: https://github.com/SeleniumHQ/selenium/blob/master/java/client/src/org/openqa/selenium/Keys.java
• Selenium Guide Book: https://seleniumguidebook.com (leading E-BOOK on using Selenium WebDriver in the primary programming languages)

• Selenium IDE: http://docs.seleniumhq.org/projects/ide/ | PROJECT | SeleniumIDE 3.x-2.x converter(this project replaces the XUL plugin with a Web Extensions API based alternative which runs cross-browser on FF 55+/Chrome/Opera, maybe Edge later... as the original XUL-based SeleniumIDE used for the past 12 years from FF 3, stopped working in FF 55+)

• flash-selenium: https://code.google.com/archive/p/flash-selenium/ | DOWNLOAD

• NodeJS Selenium: https://www.npmjs.com/package/selenium-standalone | WEB DRIVER
• flex-ui-selenium -- Selenium RC driving the tests against Flex applications: http://code.google.com/p/flex-ui-selenium/
• Selenium-Flex API - Automation for Adobe Flex applications: http://code.google.com/p/sfapi/
• Flex acceptance testing and continuous integration: http://www.blackpepper.co.uk/black-pepper-blog/Flex-acceptance-testing-and-continuous-integration.html
• Selenium in Python (WebDriver): https://pypi.python.org/pypi/selenium | DOCS

• selenium4junit: http://code.google.com/p/selenium4junit/
• Testing with PHPUnit and Selenium: http://netbeans.org/kb/docs/php/phpunit.html

jsUnit

• jsUnit -- a unit testing framework similar to jUnit but for JavaScript: http://jsunit.net/
• jsUnit project: http://jsunit.berlios.de/
• jsUnit Test Runner (example): http://jsunit.net/runner/testRunner.html?testpage=/runner/tests/jsUnitTestSuite.html
• jsUnit Documentation: http://jsunit.net/documentation/

Firebug

• FireFox F12 Dev Tools: https://developer.mozilla.org/en-US/docs/Tools (replaced Firebug as of 2016-02-08)
• Chrome F12 DevTools: https://developers.google.com/web/tools/chrome-devtools/
• IE 5-7 Debugger: https://www.my-debugbar.com/wiki/Doc/HomePage (Microsoft Script Debugger used to be the best option but DebugBar was a popular alternative, years later it is the only workable option)
• IE 8+ Developer Tools: https://msdn.microsoft.com/en-us/library/gg589512(v=vs.85).aspx
• IE - Edge -- F12 Dev Tools: https://developer.microsoft.com/en-us/microsoft-edge/tools/
• Safari : https://developer.apple.com/safari/tools/
• Opera DevTools: http://help.opera.com/Windows/9.50/en/devtools.html

FireUnit

• FireUnit: http://fireunit.org
• Unit Testing JavaScript with FireUnit: http://stephenwalther.com/blog/archive/2009/05/18/unit-testing-javascript-with-fireunit.aspx

PhantomJS

• Phantom.JS: http://phantomjs.org/
• SlimerJS: https://slimerjs.org/ (scriptable browser for Web developers)

CasperJS

• Casper.JS: http://casperjs.org/ (Navigation scripting & testing for PhantomJS and SlimerJS)
• Using CasperJS to test Picturefill (Pt.I): https://www.fourkitchens.com/blog/article/using-casperjs-test-picturefill/
• Testing JS code with CasperJS (Pt.II): https://www.fourkitchens.com/blog/article/testing-js-code-casperjs/
• Simulate user actions with CasperJS (Pt.III): https://www.fourkitchens.com/blog/article/simulate-user-actions-casperjs/
• Testing your CMS (Drupal) with CasperJS (Pt.IV): https://www.fourkitchens.com/blog/article/testing-drupal-casperjs/

JAVA

jUnit

jUnit is the leading Unit Testing framework from the xUnit family.

EasyMock

• EasyMock: http://www.easymock.org/ | CODE | DOCS
• Unit testing with JUnit and EasyMock: http://www.michaelminella.com/testing/unit-testing-with-junit-and-easymock.html

Mockito

• Mockito: http://site.mockito.org/ | CODE | [DOCS]

PowerMock

PowerMock is an extension to existing Mocking frameworks that enable testing of otherwise "untestable" areas such as:

1. final objects
2. static methods
3. private methods
4. constructors
5. enums

• PowerMock: https://github.com/powermock/powermock | SRC
• Mocks Aren't Stubs: https://martinfowler.com/articles/mocksArentStubs.html

TestNG

TestNG is a popular Java framework meant to act as a complete testing suite rather than just a basic unit testing framework like jUnit. It supports Group Testing, Testing Dependencies and Failed Test Skipping (all of which jUnit does not, without further extensions or plugins).

• TestNG: http://testng.org
• JUnit 4 Vs TestNG – Comparison: http://www.mkyong.com/unittest/junit-4-vs-testng-comparison/
• Next-Generation Testing with TestNG: http://www.artima.com/lejava/articles/testng.html
• In pursuit of code quality -- JUnit 4 vs. TestNG: http://www.ibm.com/developerworks/java/library/j-cq08296/index.html
• NetBeans IDE 7.2 Introduces TestNG: http://netbeans.dzone.com/articles/netbeans-72-introduces-testng
• TestNG Selenium: https://github.com/joshmgrant/Java-TestNG-Selenium/blob/w3c_example/src/test/java/com/yourcompany/Tests/W3CTestNG.java

t2framework

T2 is a powerful automated unit testing tool for JAVA.

• t2framework: http://code.google.com/p/t2framework/
• Automated Java testing using T2 + JUnit: http://code.google.com/p/t2framework/wiki/AutomatedTestingWithJunit

PHP

PHP Unit

• PHPUnit: http://www.phpunit.de/
• Chapter 5. The Command-Line Test Runner: http://php.find-info.ru/php/011/phpunitpg-CHP-5.html
• A short introduction to the PHPUnit test framework: http://pear.php.net/manual/en/package.php.phpunit.intro.php
• Testing with PHPUnit and Selenium: http://netbeans.org/kb/docs/php/phpunit.html?intcmp=925655

Ruby

• Watir -- Ruby library for automating tests: http://watir.com/ | DOCS (powered by Selenium/WebDriver)

Flex

• FlexMojos project (build Flex with Maven): http://flexmojos.sonatype.org/
• Flex Test Coverage Kept Simple with Flexmojos: http://www.sonatype.com/people/2010/04/flex-test-coverage-kept-simple-with-flexmojos/
• Fluint - Flex Unit and Integration Testing Framework : http://code.google.com/p/fluint/

FlexUnit

• FlexUnit: http://code.google.com/p/fluint/: http://opensource.adobe.com/wiki/display/flexunit/FlexUnit

FlexMonkey

• flexmonkey -- Flex UI Testing Tool : http://code.google.com/p/flexmonkey/

FlexCover

• flexcover -- Code Coverage Tool for Flex and AIR applications: http://code.google.com/p/flexcover/
• Code Coverage with Flex: http://eyefodder.com/

HTML

• W3C Validators - QA Toolbox: http://www.w3.org/QA/Tools/
• Browser Compatibility Check for Internet Explorer Versions from 5.5 to 8: http://www.my-debugbar.com/wiki/IETester/HomePage

C++

• UnitTest++: http://unittest-cpp.sourceforge.net/

System

Quick Test Pro

• HP's Quick Test Pro (QTP) suite: http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172122#tab=TAB3 (functional and system testing)
• Quick Test Professional (QTP) Tutorial: http://www.guru99.com/quick-test-professional-qtp-tutorial.html

Automated

• Adobe Genie - Automated UI Test tool for Adobe® ActionScript®: https://sourceforge.net/adobe/genie/wiki/Home/

Sikuli

• Sikuli: http://www.sikuli.org/ | DOCS

Performance

Taurus

• Taurus: http://gettaurus.org/ (Taurus improves integration/running/reporting on JMeter, Selenium and other leading testing tools)

LoadRunner

• HP LoadRunner: http://www8.hp.com/us/en/software-solutions/software.html?compURI=1175451
• LoadRunner - architecture overview: http://www.wilsonmar.com/1loadrun.htm

JMeter

Apache JMeter is a 100% pure Java application designed to load test client/server software (such as a web application). It may be used to test performance both on static and dynamic resources such as static files on servers, Java Servlets, ASP.NET, PHP, CGI scripts, Java objects, databases, FTP servers, and more. JMeter can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.

Additionally, JMeter can help you regression test your application by letting you create test scripts with assertions to validate that your application is returning the results you expect. For maximum flexibility, JMeter lets you create these assertions using regular expressions.

JMeter Performance Tests are stored in JMX format with the following properties (loaded in this order):

1. Thread Group
2. Config Element(s)
3. Pre-Processor(s)
4. Timer(s)
5. Logic Controller(s)/Sampler(s)
6. Post-Processor(s)
7. Assertion(s)
8. Listener(s)

The most common uses of JMeter are for Performance Testing (under typical load), Load/Stress Testing (finding maximal load), Benchmarking, Failover Testing (removing one or more "nodes" in a network to check availability/speed, etc) and Disaster Recovery (bringing nodes that were out of the Load Balancer back into service).

• Apache JMeter: https://jmeter.apache.org/ | DOCS
• JMeter - architecture overview: https://wiki.apache.org/jmeter/JMeterArchitecturalOverview
• JMeter - Component reference: http://jmeter.apache.org/usermanual/component_reference.html
• JMeter - plugins: https://jmeter-plugins.org/
• BlazeMeter JMX Converter: http://converter.blazemeter.com (HAR, SoapUI/XML, Selenium & JSON to JMeter's "JMX" format)

Penetration

• Skipfish - web application security scanner: https://code.google.com/p/skipfish/ | DOCS (released as OSS site reconnaisance tool by Google)

Resources

• OpenSource Testing Tools: http://opensourcetesting.org/
• Test Pyramid: https://martinfowler.com/bliki/TestPyramid.html
• The Practical Test Pyramid: https://martinfowler.com/articles/practical-test-pyramid.html
• Test Driven Development (TDD): http://wiki.c2.com/?TestDrivenDevelopment
• 12 Cross-Browser Testing Tools: http://www.smashingmagazine.com/2011/08/07/a-dozen-cross-browser-testing-tools/
• The Joel Test -- 12 Steps to Better Code (measure the quality of a Development Team): http://www.joelonsoftware.com/articles/fog0000000043.html
• Short, Self Contained, Correct (Compilable), Example: http://sscce.org
• Types of software Testing: http://www.softwaretestinghelp.com/types-of-software-testing/
• Pareto Chart Template: http://www.vertex42.com/ExcelTemplates/pareto-chart.html (QA assessment tool)
• Exploring Test Driven Development: http://architects.dzone.com/articles/exploring-test-driven
• Selenium WebDriver (from Java, Python, C++, etc): http://docs.seleniumhq.org/docs/03_webdriver.jsp
• User Testing Methods Comparison Chart: http://i.imgur.com/l70mclB.png
• The Big List of TDD and Unit Testing Knowledge: http://dzone.com/articles/the-big-list-of-tdd-and-unit-testing-knowledge
• Software Testing Life Cycle (STLC): http://www.buzzle.com/articles/software-testing-life-cycle.html (see STLC steps diagram)
• Software Testing Methodologies: http://www.buzzle.com/articles/software-testing-methodologies.html
• Value-Based Test Prioritization: http://dzone.com/articles/value-based-test-prioritization
• The Software Testing Wheel: http://dzone.com/articles/the-software-testing-wheel
• Keyword-driven Testing Frameworks: http://toolsqa.com/selenium-webdriver/keyword-driven-framework/introduction/
• Device/OS/Browser combinations -- How To Use Google Analytics To Find Out What Devices Your Customers Use: https://crossbrowsertesting.com/blog/development/use-google-analytics-find-devices-customers-use/
• PageObject -- abstractions for Automated Test Suites: https://martinfowler.com/bliki/PageObject.html
• Testing Podcast -- Round-up of leading Audio podcasts on Software Testing: https://testingpodcast.com/

Tutorials

• How To Use Steps Recorder -- Document full details about Computer Issues in Windows 10, 8, & 7 with Steps Recorder: https://www.lifewire.com/how-to-use-steps-recorder-2626159
• Implementing A/B Testing: http://20bits.com/articles/implementing-ab-testing/
• Selenium Tutorial for Beginner/Tips for Experts: http://jroller.com/selenium/
• Selenium IDE Tutorial – Part 1: http://www.codediesel.com/php/selenium-ide-tutorial-part-1/
• Selenium IDE Tutorial – Part 2: http://www.codediesel.com/php/selenium-ide-tutorial-part-2/
• SoapUI Pro – How to test a Web Service: http://www.joecolantonio.com/2011/12/08/soapui-pro-–-how-to-test-a-web-service/
• 10 Tips for the SoapUI Beginner: http://www.soapui.org/Getting-Started/10-tips-for-the-soapui-beginner/All-Pages.html
• Functional Testing in SOAP UI: http://www.soapui.org/Functional-Testing/functional-testing.html
• The Difference Between a FAT and a SAT: http://www.askaboutvalidation.com/the-difference-between-a-fat-and-a-sat/
• Cause-and-Effect (aka. Fishbone) Diagram: http://erc.msh.org/quality/pstools/pscsefdg.cfm
• Pareto Analysis Step by Step: http://www.projectsmart.co.uk/pareto-analysis-step-by-step.html
• Pareto Analysis - Using the 80:20 Rule to Prioritize: http://www.mindtools.com/pages/article/newTED_01.htm
• Writing Java Unit Tests: http://apps.topcoder.com/wiki/display/tc/Writing+Java+Unit+Tests
• JUnit testing EJBs in embedded Glassfish in Netbeans (the hard way): http://onthefaceofthings.blogspot.de/2013/01/junit-testing-ejbs-in-embedded.html
• New jUnit annotation "Given When Then" for testing edge cases in Java: http://java.dzone.com/articles/given-when-then-java
• Easy Mocking of Your Database: http://architects.dzone.com/articles/easy-mocking-your-database-0
• Using EasyMock or Mockito: http://www.javacodegeeks.com/2012/08/using-easymock-or-mockito.html
• Why I now appreciate testing, and why you should, too: https://medium.freecodecamp.org/why-i-now-appreciate-testing-and-why-you-should-too-74d48c67ab72
• Easy Debugging with NetBeans and XDebug: http://zurmo.org/forums/index.php?/topic/921-easy-debugging-with-netbeans-and-xdebug/
• Testing mobile sites on iPhone, iPad & iOS simulator: http://css.dzone.com/articles/testing-mobile-sites-iphone
• How to use user extensions in selenium IDE? The while loop: http://www.youtube.com/watch?v=SbN92y7JMxM
• Selenium IDE Flow Control - Goto and While Loops: http://51elliot.blogspot.ca/2008/02/selenium-ide-goto.html
• Add loops to Selenium IDE scripts: http://testingfreak.com/tools/selenium/add-loop-in-selenium-script/
• Example of "while" and "endWhile" Loop Commands In Selenium IDE With Use Of User Extension: http://software-testing-tutorials-automation.blogspot.ca/2013/07/example-of-while-and-endwhile-loop.html
• SOAP UI - Web Service Sample Project (WS mocking, inspection, functional testing & load testing): http://www.soapui.org/Getting-Started/web-service-sample-project.html
• Retry web service operations with RequestHandlerRetryAdvice: http://www.dzone.com/links/r/retry_web_service_operations_with_requesthandlerr.html
• JEMMY - Java Swing GUI testing library: http://wiki.netbeans.org/Jemmy
• How to use Jemmy with JUnit in NetBeans: http://www.theserverside.com/news/thread.tss?thread_id=78380
• jUnit for SOAP clients: http://www.jroller.com/gmazza/entry/junit_for_soap_clients
• No, you don’t need to mock your SOAP Web Service to test it: http://antoniogoncalves.org/2012/10/24/no-you-dont-need-to-mock-your-soap-web-service-to-test-it/
• Web application Debugging. Java-Web, Servlets and JSP 7 (SCREENCAST): http://www.youtube.com/watch?v=XdjGQVav10M
• Writing Clean Tests (from perspective of a Java dev): http://www.petrikainulainen.net/writing-clean-tests/
• Verifying your Web Applications Works - From Unit Testing to End-To-End Testing: http://java.dzone.com/articles/verifying-your-web
• Mobile WebApplication Testing (REFCARD): http://refcardz.dzone.com/refcardz/mobile-web-application-testing
• 10 Steps To Kick-Start Your Automated Mobile Testing: http://www.dzone.com/articles/10-steps-kick-start-your
• Maven Test Isolation: http://java.dzone.com/articles/maven-test-isolation
• Introduction To JavaScript Unit Testing: http://www.smashingmagazine.com/2012/06/introduction-to-javascript-unit-testing/
• Using variables with Selenium IDE: http://nickbartlett.com/using-variables-with-selenium-ide/
• Storing variables in Selenium IDE: https://webigniter.wordpress.com/2011/04/07/storing-variables-in-selenium-ide/
• How to store the content/value of xpath?: http://stackoverflow.com/questions/11010992/how-to-store-the-content-value-of-xpath
• (Unit Testing) Why and what is Arrange Act Assert?: https://web.archive.org/web/20150307051201/http://www.arrangeactassert.com/why-and-what-is-arrange-act-assert
• Stop Waiting & Start Testing With Open Source Tools: http://info.blazemeter.com/stop-waiting-start-testing-open-source-tools-ty
• A Concise Guide to Knowing What, When, and How to Load Test: http://dzone.com/articles/how-much-load-can-your-site-handle-a-concise-guide
• jMeter Intro: http://2min2code.com/articles/jmeter_intro
• A Load Tester’s Guide to JMeter & BlazeMeter: https://blazemeter.com/blog/load-tester%E2%80%99s-guide-jmeter-and-blazemeter
• Application (and Mobile) Quality Maturity: http://docs.perfectomobile.com/docs/resources/infographic/5-stages-of-mobile-quality-maturity-ig.pdf
• Effective and Affordable Load Testing Using JMeter: http://omerio.com/2011/05/22/effective-and-affordable-load-testing-using-jmeter/
• Introduction to JMeter: http://www.guru99.com/introduction-to-jmeter.html
• JMeter Ant Task: http://www.programmerplanet.org/projects/jmeter-ant-task/
• JMeter Tutorials Series – Learn Load & Performance testing using JMeter in 11 days: http://www.softwaretestingclass.com/learn-jmeter-performance-testing-jmeter-tutorial-series/
• How to use Timers in JMeter: http://www.guru99.com/timers-jmeter.html
• How to use Controllers in JMeter: http://www.guru99.com/controllers-in-jmeter.html
• How to Convert SoapUI XML to JMeter JMX: https://dzone.com/articles/how-to-convert-soapui-xml-to-jmeter-jmx
• SpiraTeam, the Simple Alternative to HP ALM: https://www.inflectra.com/SpiraTeam/Highlights/HP-ALM-vs-SpiraTeam.aspx
• Getting Node.js Testing and TDD Right: http://blog.codeship.com/getting-node-js-testing-and-tdd-right/

External Links

• wikipedia: Category:Software testing
• wikipedia: Test plan
• wikipedia: Test method
• wikipedia: Test Management Approach (TMap)
• wikipedia: Manual tests
• wikipedia: System testing vs acceptance testing
• wikipedia: Enterprise risk management (ERM)
• wikipedia: Swiss cheese model
• wikipedia: Physical test
• wikipedia: Pair testing
• wikipedia: All-pairs testing
• wikipedia: Software testability
• wikipedia: Web testing
• wikipedia: Web server benchmarking
• wikipedia: Game testing
• wikipedia: Test automation
• wikipedia: Test-driven development
• wikipedia: Quality control
• wikipedia: Six Sigma (branded methodology & certification label for Software Quality & Testing)
• wikipedia: Verification and validation
• wikipedia: Formal verification
• wikipedia: Proof of concept (commonly abbreviated PoC)
• wikipedia: Software release life cycle
• wikipedia: Independent test organization (preference to have a neutral, non-invested 3rd party to do all or specific - i.e. Performance - phases of testing of a product/service/system)
• wikipedia: Seven Basic Tools of Quality
• Software Quality at Top Speed: http://www.stevemcconnell.com/articles/art04.htm
• Professional Quality Assurance (PQA): http://www.pqatesting.com/
• Can You Explain a Sample JUnit Test Case Class?: http://sqa.fyicenter.com/FAQ/JUnit/Can_You_Explain_a_Sample_JUnit_Test_Case_Class_.html
• Mocks Aren't Stubs (the differences explained): http://martinfowler.com/articles/mocksArentStubs.html#TheDifferenceBetweenMocksAndStubs
• Why Evangelising Unit Testing Needs to Stop: http://java.dzone.com/news/why-evangelising-unit-testing-
• Testing Your Flex Applications by Michael Labriola on Adobe TV: http://tv.adobe.com/watch/max-2008-develop/testing-your-flex-applications-by-michael-labriola/
• DAO Refactoring -- New unit tests: http://blog.restafarian.org/2009/04/dao-refactoring-new-unit-tests/
• Can your Selenium do that? Testing flash/flex and silverlight in web apps with iMacros: http://www.pathf.com/blogs/2009/04/can-your-selenium-do-that-testing-flashflex-and-silverlight-in-web-apps-with-imacros/
• Ajax Testing -- Doubling Down with Selenium and JMeter: http://www.pathf.com/blogs/2008/04/ajax-testing-do/
• CodeCoverage Plugin Tutorial: http://wiki.netbeans.org/TutorialCodeCoveragePlugin?intcmp=925655
• How To Ask Questions The Smart Way: http://www.catb.org/~esr/faqs/smart-questions.html
• What is "Done"?: http://kaczanowscy.pl/tomek/2010-05/what-is-done
• Evil Unit Tests: http://www.javaranch.com/journal/200603/Journal200603.jsp#a1
• JUnit + Jtest = Automated Test Case Design and Static Analysis: http://www.devx.com/Java/Article/15625/1954
• Does Jtest make the grade?: http://www.javaworld.com/jw-03-2001/jw-0323-lw-jtest.html?page=1
• Using the Embedded EJB Container to Test Enterprise Applications: http://netbeans.org/kb/docs/javaee/javaee-entapp-junit.html?intcmp=925655
• We Can’t (reliably) Measure Anything in Software Development: http://java.dzone.com/node/130159/
• End-To-End Developer Testing: http://www.drdobbs.com/tv/1813092199001
• The Secondary Effects of Automated Testing: http://www.drdobbs.com/tv/1809012403001 (author of "Working Effectively with Legacy Code")
• How cool is integration testing with Spring+Hibernate: http://dinukaroshan.blogspot.ca/2012/11/how-cool-is-integration-testing-with.html
• Java 7 Code Coverage for NetBeans: http://netbeans.dzone.com/articles/jacocoverage-java-7-code
• Can your Selenium do that? Testing flash/flex and silverlight in web apps with iMacros: http://pathfindersoftware.com/2009/04/can-your-selenium-do-that-testing-flashflex-and-silverlight-in-web-apps-with-imacros/
• Excuses For Not Testing: http://java.dzone.com/articles/excuses-not-testing
• What Do Software Defects Really Cost? The $2.3 Billion Bug: http://blog.parasoft.com/cost-of-software-defects
• Decorator Design Pattern in Automation Testing: http://dzone.com/articles/decorator-design-pattern-in-automation-testing
• The Testing Center of Excellence is Dead: http://dzone.com/articles/the-testing-center-of-excellence-is-dead
• Notes on Testing Java EE: https://www.youtube.com/watch?v=x4Fxur-0SB0
• Selective Unit Testing – Costs and Benefits: http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/
• JUnit - Testing Private Methods and Fields: http://markreddy.ie/junit-testing-private-methods-and-fields/
• Integrating SOAP UI with JUnit: http://www.soapui.org/Test-Automation/integrating-with-junit.html
• Unit-Testing of Web-Services with JUnit - SOAP Services: https://www.paluch.biz/blog/53-unit-testing-of-web-services-with-junit-soap-services.html
• Testing Web Services from JUnit using SoapUI: http://opensourcesoftwareandme.blogspot.ca/2012/10/testing-web-services-from-junit-using.html
• Software Test Automation with Java, PhantomJS and Selenium: http://rumandrye.wordpress.com/2013/03/12/software-test-automation-with-java-phantomjs-and-selenium/
• When Do You Stop Testing?: https://dzone.com/articles/when-do-you-stop-testing
• What’s the Right Ratio Between QA Testers and Developers?: http://dzone.com/articles/whats-the-right-ratio-between-qa-testers-and-devel-2
• iOS 10 Is About to Disrupt Mobile Testing Plans: https://dzone.com/articles/ios-10-is-about-to-disrupt-mobile-testingnbspplans
• Mobile Test coverage: https://dzone.com/storage/assets/4249372-perfecto-device-magazine-the-digital-test-coverage.pdf
• Diminishing Returns in software development and maintenance: http://swreflections.blogspot.ca/2011/11/diminishing-returns-in-software.html
• Test Pyramid: https://martinfowler.com/bliki/TestPyramid.html
• What is the Right Ratio of Software Developers to Testers?: https://www.softwaretestpro.com/what-is-the-right-ratio-of-software-developers-to-testers/
• Just Say No to More End-to-End Tests: https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html
• Xray for Jira - How to automate your QA process: https://www.slideshare.net/XpandIT/xray-for-jira-how-to-automate-your-qa-process
• Software Defects - Do Late Bugs Really Cost More?: https://developers.slashdot.org/story/03/10/21/0141215/software-defects---do-late-bugs-really-cost-more
• The Cost of Fixing Bugs in Production: https://www.genrocket.com/the-cost-of-fixing-bugs-in-production/
• Quantifying The Cost of Fixing vs Preventing Bugs: http://www.newelectronics.co.uk/article-images/65147/Vector_PDF.pdf
• The True Cost of a Software Bug -- Part I: http://blog.celerity.com/the-true-cost-of-a-software-bug
• Examining the Agile Cost of Change Curve: http://www.agilemodeling.com/essays/costOfChange.htm
• Why You Should Stop Testing Every Class NOW: https://dzone.com/articles/why-you-should-stop-now-testing-every-class
• What Is Refactoring? (and Testing's role within it): https://dzone.com/articles/what-is-refactoring
• Two years with no testers -- What I learned: https://techbeacon.com/two-years-no-testers-what-i-learned
• Top 10 Testing Frameworks and Libraries for Java Developers: https://dzone.com/articles/10-essential-testing-tools-for-java-developers
• Modularity and testability: http://www.codingthearchitecture.com/2014/10/01/modularity_and_testability.html

See Also

Ant | Maven | CI/CD | SCM | Logging | Monitoring | Scheduling | Productivity | Security