At QApp we take extra care of testing our products to make sure they are as reliable as possible

Code analysis

Format check
Each push comes with an automated format check, ensuring code style consistency.
Tools: clang-format, autopep8.

Static analysis
Automated bug search that does not require actually running the code and helps improve the code quality at early stages.
Tools: clang-tidy.

Code-review
At least two team members are involved in code-review.

Code review by security experts
Security experts (ASE) occasionally review the code. Detected vulnerabilities are studied and patched at developer workshops. Detected vulnerabilities are ranked on a four-point scale (Critical, High, Low, None). Patching Critical- and High-level vulnerabilities is the main priority.

Functional testing

Unit testing
Both external interface and internal implementation are tested. Final code coverage is over 90%.
Tools: Googletest, gcovr.

Unit testing with sanitizers
The test uses a set of GCC and CLang compiler-based sanitizers. The code is instrumented with additional instructions at the compilation stage and then run in a controlled environment. Among other things sanitizers help detect:

• Memory errors.
  
• Memory leaks.
  
• Data races.
  
• Overflow errors.
  
• Initialization errors.
  
• Type errors.
  
• Invalid arithmetic operations.
  
• etc.

Tools:

• CLang: AddressSanitizer, MemorySanitizer, ThreadSanitizer, UndefinedBehaviorSanitizer.
  
• GCC: AddressSanitizer, UndefinedBehaviorSanitizer.
  
  

Integration testing
Comprehensive external function testing (API) that helps recognize:

• Boundary values.
• Invalid input data.
• Changes in intermediate data.
• Different entropy source porting.

Integration test documentation is included directly in the code comments. Scenarios and parameters are described using doxygen tags. Integration testing provides ready test documentation in BDD format.
Tools: Googletest, Doxygen

End-To-End testing
A final package containing all the necessary software and documentation is prepared for the customer. Each package is automatically tested in a clean testing environment. We check the following things:

1. Package name
   
2. Package contents
   
3. Package setup
   
4. Usage examples
   
5. Documentation
   

Tools: pytest.

Integration testing with other products
We link the product to specific versions of dependent products, such as a tag name, or at most, a branch name. Ready-made CI artifacts are used as well. Thus, the testing runs with already collected artifacts. As a result, we get continuous integration.
Tools: pytest

System testing

Performance testing
Algorithm performance testing allows for:

1. Tracking performance dynamics from version to version.
   
2. Comparing the current implementation to alternative ones.
   
3. Calculating the time it takes to process different input data, thus, testing the algorithm's resistance to timing attacks.
   

Tools: Google.Benchmark, SUPERCOP.

Fuzzy-testing
Fuzz testing helps estimate, how robust a software is against random or invalid input data. Fuzz testing provides:

1. 1. A set of files comprising all the detected code paths. This set can be further used in testing for timing attacks.
   2. A set of files that caused the software to freeze or crash.

Tools: AFL (American Fuzzy Lop).

Timing attack-resistance testing
One of the main types of testing for cryptographic software. There are two potential strategies:

1. Developing a native framework that estimates the runtime statistics based on the fuzzer's input data.
   
2. Integrating dudect – a specialized tool for estimating the code runtime statistics.
   

Tools: native framework, dudect.

Security audit
Scanning for vulnerabilities using specialized static and dynamic analysis tools.
Tools:

1. Dynamic analysis: GDB/GDBSERVER, GEF
   
2. Static analysis: IDA, Ghydra, Unicorn
   
3. Decompilation and dynamic instrumentation for Android applications: JEB, JADX, Frida
   

CI

CI is at the core of the entire development process. All automated tests, as well as package assembly and validation, are performed in CI..
Tools: GitLab.

Assembly levels
There are several assembly levels, which regulate:

1. 1. A set of configurations to be tested.
      
   2. Implementation of the most demanding Jobs. For example, performance testing and timing attack-resistance testing

The fastest surface level is for feature-branch assembly and testing:

The most time-consuming and comprehensive level is for master-branch updates and nightly builds:

We have also added a manual-Job set, where you can manually specify, which Job you would like to run:

Nightly builds
The nightly master assemblies are performed every day. We use the maximum set of configurations.

Workflow development

Our team relies on a simple, but efficient Workflow: