Software / System Integration: Tips and Best Practices
For software developers working for any medium-size or large company, dealing with different systems is daily routine.
System integration is the key to enable communication between different applications and services, it helps us to offer extended services to users, automation, and all-in-one solutions.
In this article I describe several considerations to keep in mind when integrating systems.
PREPARATION
1. Understand the business requirements: for each new development I like to answer the main 5 questions: what, why, how, who, when?
For well defined projects, a project charter can answer these questions, this is like a one page overview of the project.
2. Analysis of components to be integrated:
- Type of system; open source or proprietary software
- How can you connect to these systems; network configuration (access, speed) and protocols
- Limitations: size of data, time slots to query, processing time for complex algorithms, number of queries / users
3. Analysis of the tool to enable integration: often, system integration is about building an Application Programming Interface (REST APIs) or Middleware, unless the software has its own interfaces.
4. Some considerations to prepare for development:
- Hardware / software requirements, if any
- Programming languages, for APIs the common ones NodeJS, .NET, Python. This depends on what adapt better to your needs and the environment
- Data formats, ETL processing required, data governance
- Design a workflow with tasks that components should perform; a sequence and a components diagram give a high-level overview
Here an example of a complete services infrastructure I developed for my company:
5. Before start coding, keep your stakeholders informed
Keeping stakeholders informed about this new system integration will help you to gain continuous support and avoid any inconvenience, remember to point out pros / cons, limitations, timing.
At this point you can create an Integration Planning document with all the information collected.
DEVELOPMENT
Either if you opted for developing a new API or you are using the same software, here some considerations:
1. Installation of Hardware and Software components if any
2. Setup production and development environment
3. During the development consider:
- Check data format received and parse into models
- Add try / catch, especially for external connections
- Log errors (database or file logs), and
- Set notifications in case of failure (emails to administrator)
4. For big data or heavy processing requirements; consider implementing multi-thread or parallel processing with request queues
5. Test.. test and test
- Perform individual tests between subsystem before testing all together
- Try possible edge case scenarios, anything that can break the system
- Perform stress test (large scale systems)
6. Define when the tool is triggered
- Pooling: API is scheduled to make calls each x time, independent if there are changes or not
- Webhook: API receives calls only when there are changes
DEPLOYMENT
Once development and testing finished, inform stakeholders and move to production
1. During the first days, keep monitoring system processes, make sure it works as expected
2. Ensure the failure detection and notifications are working, and plan accordingly all fault tolerance measures (what-if and edge cases)
3. Setup backup plans; when dealing with data changes or possible data loss
4. Prepare technical documentation, include diagrams to facilitate the overview of different components integrated
Happy coding!
