Abort the job on the main thread if the test suite fails (#32)

* feat(suite)!: Stop on fail in Suite.

* feat(tests): More tests

More tests and another suite case have been added to test the different RUN_TESTS options.

* fix(suite): Fixed errors that prevented compiling.

Bugs that prevented compilation have been fixed. ENUM TestRunBehavior names have been changed and relevant documentation has been added.

* feat: Added default behavior un RUN_TESTS

* fix: 🐛 Fixes in conditionals of enum values.

* refactor: Use of default value on RUN_TESTS

* feat(suite): Added output on FreeTest.

An output has been added in the FreeTest in case of error and with behavior "ABORT_ALL_ON_FAIL".

* refactor(suite): Output for freetest.

Se ha movido y modificado el output de error en caso de error en freetest con el modo 'ABORT_ALL_ON_FAIL'. Se ha añadido output de warning en caso de error en freetest con el modo 'HALT_SUITE_ON_FAIL'.

* refactor(suite): 🎨 New function for suite process.

The code in which the suite tests were processed has been moved to its own function.

* refactor: 🚧 Added tests to validate suite logic.

* feat(suite): Throw exception on test error

In case of an error in the tests an exception is thrown. This exception is thrown at different times depending on the error mode used.

* feat: Added output for halted suite

Added output when a test fail on a suite on HALT_SUITE_ON_FAIL mode. Reformat file.

* fix: 🐛 Exit on default mode

Fixed a bug on ABORT_ALL_ON_FAIL mode when a test fails in a suite. Added output.

* refactor: Default on RUN_TESTS()

Using Default value on RUN_TESTS(). Added comments.

* chore: Added .clang-format

* refactor: Substitution of exception by exit(1)

The exception is eliminated in favor of exit(1) since error output is already returned.

* docs: Docs for TestResults.

* docs: Docs for checkForTestErrors.

* docs: Documentation for test functions.

* docs: 📝 Documentation update to reflect the modes.

The test suite documentation has been updated, adding information on test execution modes. Updated the example of use with more tests, to be able to see more clearly how the modes work in the result images.

* chore: 🔥 An obsolete image is deleted.

The old example image of the output of the previous example code in the README is removed, as it does not reflect the actual code written and the modes are not reflected.

* docs: 📝 Removed excess code in example

* docs: 📝 Information on modes is added.

Due to an oversight this information had not been added in one of the previous commits.

* docs: 📝 Added test suite reference.

Author: gbm25

.gitignore

@@ -10,3 +10,4 @@ compile_commands.json
 notes.txt
 *.exe
 .cache
+.clang-format

README.md

@@ -14,6 +14,7 @@ Quick description, and links to the documentation of all the libraries of the pr
 - [Text](./zero/ifc/text/README.md) - Offers utilities for text manipulation, string splitting, formatting, styling, and print to console functionalities.
 - [Types](./zero/ifc/types/README.md) - Information about types and traits for types
 - [Math](./zero/ifc/math/README.md) - Mathematical operations and functions
+- [Test Suite](./zero/ifc/test-suite/README.md)  - A flexible and simple test-suite.
 
 ## Sponsorship
 
@@ -48,16 +49,6 @@ system in a real world environment, so people can profit by taking examples of h
 in a big scale project. Also, we introduce the changes of the latest releases available here, so it serves as well as
 a kind of latest up-to-date guide.
 
-## Testing tools
-
-We are using [`catch2`](https://github.com/catchorg/Catch2) as the testing suite for `Zero`, in its *header-only* version.
-
-We made a try to replace it with `boost::ut`, that spots that is compatible with modules, but we are getting all kind of errors in the process of convert the provided header to a precompiled module, so we will stay
-with `catch2` for now.
-
-Also, we are looking for generate a precompiled header of the Catch's suite in order to decrement
-the tests compilation time.
-
 ## General guidelines and API design
 
 This entry is a collection of both the things that we're looking for, or focusing on

zero/assets/tsuite_results.png

@@ -65,6 +65,21 @@ Test cases within a suite can be registered using the `TEST_CASE(...)` function,
 just like standalone tests, but choosing the overload that receives as its first parameter
 a reference to the test suite.
 
+### **Test Modes**
+In the pursuit of flexibility and control over the test execution process, our testing framework offers different modes of operation. These modes determine the test runner's behavior in response to test failures, allowing users to tailor the testing process according to their specific needs or the nature of the tests being run.
+
+1. **CONTINUE_ON_ERROR**
+   - **Description**: This mode ensures that all tests are executed regardless of any failures encountered. It's ideal for comprehensive error analysis or understanding the full extent of issues in the codebase.
+   - **Use Case**: Opt for this mode when a complete analysis of the system's state is required, particularly to identify all potential problems in a single test run.
+
+2. **HALT_SUITE_ON_FAIL**
+   - **Description**: In this mode, the test suite halts the execution of the current suite or free tests upon encountering a failure, then proceeds to the next suite or free test. It allows for a quick bypass of problematic tests while still performing remaining tests.
+   - **Use Case**: Useful for scenarios where quickly identifying and addressing failures is important, without getting bogged down by tests in a problematic suite.
+
+3. **ABORT_ALL_ON_FAIL (Default mode)**
+   - **Description**: This mode adopts a zero-tolerance approach towards test failures. As soon as any test fails, it immediately halts all further testing activities.
+   - **Use Case**: Use this mode to avoid wasting time on further tests when there's already a known issue that needs fixing.
+   
 ## **Example of usage**
 
 Here's an example of how to use the custom test-suite to write and run test cases:
@@ -76,52 +91,104 @@ import std;  // Should be ready on all the major compilers for C++23. But until
 // with the `Zork++` out of the box solution based on Clang modulemaps.
 import tsuite;
 
-// Define test functions. Should be void functions that later w'd be registered in a suite.
+// Let's define some example test functions using the assertion function
 void testAddition() {
     int result = 2 + 2;
     assertEquals(4, result);
+    assertEquals(4, result);
+    assertEquals(4, result);
+}
+
+// Let's define some more example test functions using the assertion function
+void testSubtraction() {
+    int result = 3 - 2;
+    assertEquals(1, result);
+    assertEquals(23, result);
+    assertEquals(1, result);
+}
+
+
+// Let's define even more example test functions using the assertion function
+void testMultiplication() {
+    int result = 2 * 2;
+    assertEquals(4, result);
+    assertEquals(4, result);
+    assertEquals(4, result);
 }
 
+
 // Passing two pointers to compare if the values that they point to are equals
 void testPtrsAddition() {
     int result = 2 + 2;
     int expected = 4;
+    int wrongExpected = 16;
     assertEquals(&expected, &result);
+    assertEquals(&wrongExpected, &result);
 }
 
 // Driver code
 int main() {
-    // Free tests cases registration examples
-    
+
+    TEST_CASE("Multiplication Test", []() {
+        int result = 5 * 3;
+        assertEquals(15, result);
+        assertEquals(15, result);
+    });
+
+
     // Register a new test case using a function pointer.
     TEST_CASE("Addition Test With Pointers", testPtrsAddition);
-    
-    // Users can register a new test case using lambdas, avoiding to write standalone functions
+
+    // Users can register a new test case using lambdas, avoiding writing standalone functions
     TEST_CASE("Subtraction Test", []() {
         int result = 5 - 3;
-        assertEquals(122435, result);
+        assertEquals(2, result);
+        assertEquals(2, result);
     });
-    
+
     // Registering test cases into test suites, to group and relate tests that makes sense to exists
     // as a part of a whole
-    
+
     // Instantiate a new test suite, giving it a unique identifier.
     TestSuite suite {"My Suite"};
     // Register test cases using function pointers into a test suite
     TEST_CASE(suite, "Addition Test", testAddition);
-    // Force a warning that alerts the user that the test will be discarded, since already
+    // Forces a warning that alerts the user that the test will be discarded, since already
     // exists one with the same identifier in the given suite
     TEST_CASE(suite, "Addition Test", testAddition);
-    
+    // Register a test case designed to fail, useful for testing the behavior 
+    // of RUN_TESTS with different failure modes.
+    TEST_CASE(suite, "Subtraction Test", testSubtraction);
+
+    // Register additional test cases to verify the functionality of RUN_TESTS
+    // under different conditions.
+    TEST_CASE(suite, "Multiplication Test", testMultiplication);
+
+    // Create another test suite to further validate the behavior of RUN_TESTS
+    // with multiple suites, especially under different failure modes.
+    TestSuite anotherSuite {"Another Suite"};
+    TEST_CASE(anotherSuite, "Addition Test", testAddition);
+    TEST_CASE(anotherSuite, "Subtraction Test", testSubtraction);
+    TEST_CASE(anotherSuite, "Multiplication Test", testMultiplication);
+
     // Don't forget to call this free function, to run all the tests written!
+    // Options are: CONTINUE_ON_ERROR, HALT_SUITE_ON_FAIL, ABORT_ALL_ON_FAIL (default)
     RUN_TESTS();
 
     return 0;
 }
 ```
 
-With these example, you will see this result:
-![img.png](../../assets/tsuite_results.png)
+With these examples, you will see different results depending on the mode used:
+
+1. **ABORT_ALL_ON_FAIL** mode (Default):
+   ![Abort All on Fail Mode](../../assets/tsuite_results_ABORT_ALL_ON_FAIL_mode.png)
+
+2. **HALT_SUITE_ON_FAIL** mode:
+   ![Halt Suite on Fail Mode](../../assets/tsuite_results_HALT_SUITE_ON_FAIL_mode.png)
+
+3. **CONTINUE_ON_ERROR** mode:
+   ![Continue on Error Mode](../../assets/tsuite_results_CONTINUE_ON_ERROR_mode.png)
 
 ## Funny facts