During the course of the project, it became evident that the growing scale of the business and expectations to control and react to changing business environment require the company to gather data, analyse it and make decisions based on insights rather than intuition.
The first phase of the project concentrated on understanding the needs of the organisation as a whole and the needs of each particular department. Based on the acquired knowledge, areas to cover were indicated and technological solutions to choose from were shortlisted. As a result of multiple internal workshops, a decision was made to focus on Microsoft and open-source solutions. The company has used Office365 as a major business tool for several years already and their priority was to minimise the impact on users by sticking to solutions that they are familiar with. Microsoft Azure was chosen as a cloud platform for the development of applications and processing of the data, as well as PowerBI for data visualisation purposes. The GUI for the data lake as well as the data lake itself were based on open-source solutions and built from scratch.
In the next stage of the project, the data environment of the customer was mapped. We analysed data to see if APIs or other interfaces were allowing for data collection and if they fulfilled business requirements.
An arduous process of data collection began based on a complete list of data sources, data types to be collected, and available interfaces. A lot of data is collected using open-source-based software robots, which trawl through websites and gather data. The next step was to implement BRAINHINT Robot Farm, which enables orchestration of dozens of robots to ensure completeness and credibility of the data.
The last step was to roll out the technology for users. Data lake entered the production stage and was integrated with all of the data sources, such as the CRM system, publicly available websites and databases, call centre systems, and PowerBI to present the data. Additionally, we developed a GUI for data lake, which allows users to export data using an interface similar to Excel pivot tables. All systems were made available based on SSO to meet the security requirements of the organisation. Our work continues to provide department-specific functionalities.
