Technical overview
Explore the detailed technical foundation of SDK.finance to understand, deploy, and manage the Platform effectively. This guide covers everything from architecture fundamentals and data management to source code handling, scalability, operations, and security.
Architecture overview
See the schemes below to understand the high-level design of the system, including components, modules, and their interactions.
High-level architecture
Deployment model
Schematic diagram of the environment
Non-high load
High load
Template for disaster recovery / fault-tolerant architecture
Parameters for minimal hardware configuration
Please see both minimal and recommended hardware parameters.
System architecture and resilience
Impacts of failure
The pre-production and production environments are isolated, so a failure in one does not directly impact the other. This setup allows for testing and bug-fixing stages without affecting the live environment.
Application server side
The system is stateless, and all activity is mapped on an ACID type database. Both MongoDB and PostgreSQL used in SDK.finance are ACID compliant. If an application server goes down, the state of the system is always available for another application server copy.
The application server and DB copy must be supported with an Active-Active type of “Disaster Recovery” infrastructure, where the DB and application servers are continuously synchronized in real-time from the DB replica node.
With high traffic, the application server can operate as a cluster of several application server instances behind a load balancer. In case one of the cluster nodes goes down, the remaining traffic is distributed to the other cluster nodes.
Database management
Each user’s database is isolated and managed by the user. This isolation means a failure in one user’s database does not impact others. DB management is also a part of the daily duties of the DBA (database architect) position in the company and relies on the available toolset. Some tools are available only for clouds, while others can be deployed in on-premises data centers. Continuous monitoring, backups, and data-retention policies are daily duties of the IT support team.
API gateway
We are using an API-first approach and all features are available via APIs in SWAGGER (Open API 3.0.3) format. The latest version is always available here, updated daily: SDK.finance API reference
Thanks to an API gateway used to manage requests, if one service fails, the gateway can route traffic to fallback services or another node in the cluster.
In the SDK.finance system, we have implemented an exception-handling mechanism designed to ensure continuity of operations in the event of an error. This mechanism isolates errors that occur in specific processes or during the execution of individual operations. By effectively catching and managing exceptions locally, the system prevents these errors from impacting the entire system’s stability. This allows other operations to continue functioning normally, thereby enhancing the overall resilience and reliability of our application. Additionally, the API orchestration layer can be extended with third-party tools that operate as a high-level API management tool plus an additional security layer. We recommend using KONG as a tool we have successfully used in production.
Database schema documentation
All system activity is mapped automatically in the DB via Hibernate framework. We are using a code-driven DB approach. Its advantages are listed below:
Enhanced performance
Using the code-first approach, developers can optimize the database structure based on the application’s needs rather than being confined to a preset schema. This allows for more efficient use of database resources, such as creating indexes and optimizing queries based on the actual data usage by the application.
Source code maintenance
The code-first approach facilitates source code maintenance as changes to the data model can be implemented directly in the code, without switching between different tools or languages (like SQL). All changes are centralized in the code, simplifying version control and change management.
Adding new modules
Expanding the application with new functional modules becomes easier, as new entities or changes to existing entities can be seamlessly integrated into the overall database structure. Code-first allows for automatic database migrations, significantly speeding up the development process and simplifying the integration of new components. It also significantly reduces the time for DB management and CI/CD pipelines for updates.
Overall, the code-first approach offers flexibility and manageability in development, greatly simplifying the processes of scaling and adapting the system to changing business requirements.
Testing documentation
Documentation related to testing, including test plans, test cases, and test results. This is to ensure the reliability and quality of the codebase.
Currently, we have:
- Collections in Postman
- Test cases writing in TestRail
- Unit tests in code. Application functionality is covered by numerous unit and integration tests.
- For unit and integration testing, we use:
Monitoring and logging documentation
We are using an integrated Java Spring engine Actuator for low-level monitoring. Additionally, we recommended that our clients use a more advanced framework for overall system performance, health check monitoring, and alerting like:
Or available from Infrastructure providers, for example:
AWS
MS Azure
Security documentation
Documentation related to security measures implemented in the application, including authentication mechanisms, authorization policies, data encryption, and vulnerability assessments.
Secure Software Development Standard:
- Auth tools: Native JWT, OAUTH 2 Native or SSO via Keycloak
- Application encryption: SHA 2.1, HmacSHA256
- Code quality control: LLMs with IDE (JetBrains AI Assistant, Amazon Code Whisper(Q) and Git Co-pilot) and manual code review
- Static code analysis: SonarQube or JetBrains Qodana.
Check SDK.finance system performance testing to explore the Platform workload capacity.
Single points of failure and mitigation
Database server
If the database server fails, it can bring down the entire application. As mentioned before, DBs must continuously have a backup stage for long-term storage, plus the “Active-Active Disaster Recovery” stage that intercepts transactions almost in real-time. If the main DB goes down for some reason, internal mitigation with the IT support team must include setting up database replication, backups, and failover mechanisms.
API gateway
If the API gateway fails, all services dependent on it will be impacted. Using multiple instances of the gateway and load balancing can mitigate this risk, as mentioned earlier, or with a switch to the Disaster Recovery stage.
Application server
A failure in the application server can halt all processing. Running multiple instances and using load balancers can distribute traffic and avoid a single point of failure, or with a switch to the Disaster Recovery stage.
Adding, removing, and scaling modules
Removing or adding new modules
Each module has its own DB migration scripts, but tables are not independent of each other. Some fields have references (Private Key, Foreign Key) to tables that have mapped entities in other modules.
If you want to decouple the module from the core application, these relationships should be removed (additional migrations and changes in the codebase), but it’s doable after further discussion.
Scaling up a single module of the application to accommodate increased user traffic or data growth for that module
The module is a part of the project until it is refactored to a stand-alone service. If we need more copies of the module or service, we just run more instances, and the load balancer sends traffic of certain API calls to a cluster of instances. It’s easy to do with the KONG API orchestration solution we mentioned above.
Integrating current projects with other microservices with their data storage
The SDK.finance platform is designed with flexibility in mind, allowing for easy integration with other microservices. Each service can maintain its data storage, ensuring seamless and efficient integration. Other microservices can easily communicate with SDK.finance modules or services by using an Enterprise Message Bus such as Kafka which SDK.finance supports out of the box. Alternatively, it can be integrated with similar services like RabbitMQ, Amazon MSK, Oracle Streaming and others.
Codebase complexity
The modular design of the codebase ensures that it remains manageable as the application grows. Each module is self-contained, making it easier to understand and maintain. This approach reduces the complexity of the overall system.
Monitor the health and performance of various parts of the application individually
In the Java Spring Framework, there is an Actuator module available for use in the project. This tool helps monitor and manage the application in production at the lowest tech layer with multiple metrics (https://www.baeldung.com/spring-boot-actuators). Additionally, business-related monitors, log representation, and alerting can be done with multiple available tools. We recommend NewRelic if it’s allowed, and the ELK stack for on-premises data centers with a free edition.
The leniency of the selection of the database engines with alternative DBMSs (MySQL or Cassandra, etc)
In the application, Spring Data JPA allows implementing communications between applications and different types of DBMSs. We also use Hibernate, which minimizes the impact on developers and abstracts data manipulation with any of the popular DBs of communication dialects.
Source code repository
Currently, we are using GIT with GitLab. For source code transfer and updates, we provide two ways:
- Pull, your repository sync from our side
- Push, we deliver updates directly to your repository