Head of Findwise Finland
Challenge
The National Institute for Health and Welfare Finland (THL) studies, monitors and develops measures to promote the well-being and health of the population in Finland. They gather and produce information based on research and statistics, and provide expertise and solutions that stakeholders can use to support decision-making and other work.
In other words, THL collects a lot of data. A bunch of online forms, integrations with other systems, research, analysis etc. All of those systems need to be up and running, and all of them create a lot of logs. There are access logs, error logs, standard application logs and logs from custom applications. Some logs are even stored within databases or in some legacy systems.
In the end it’s really just down to having one centralised log management system.
Solution
At Findwise, part of Tietoevry Create, our experts have noticed an interesting re-occurring scenario the last couple of years when working with Elastic. One common way organizations implement the popular open source tool is that someone just starts using Elastic, others latch on and then it grows. It is not uncommon for us to start working with a client when they already are in production and are having challenges due to unmanaged scaling.
At THL it was different – they immediately felt the need for a central solution and started planning it as such. This allowed us to ensure a production ready and scalable solution right from the start.
The need for a centralized log management system came due to many reasons but most importantly due to wanting to:
- Get an overview of how systems are working
- Easily spot problems and do troubleshooting
- Work around security-related matters
The solution is a somewhat typical (modern) Elastic stack installation. Data is fetched using Beats. They ship logs to a centralized Logstash installation. Logstash handles the processing of the data into suitable and agreed format and stores all data in Elasticsearch. Kibana is used for visualizing, analyzing and managing the data. THL also uses Elastic commercial features (previously known as X-pack), among others, for securing the whole environment.
Result
Here is a typical situation: something isn’t working and THL needs to understand what and why. Using Kibana dashboards, they can now very quickly see where the problem is and when an error occurred. The dashboards welcome possibilities to drill down to root problems or just inspect usage. And all this, without first needing to SSH into some system and localizing the correct logs to look at.
Log management and Kibana also provides clear statistics on when and how systems are being used. Basic stuff when it comes to log management in general but for THL is a huge improvement from before.
So in essence it is going from the console, grepping and tailing logs..
.. to an intuitive and easy to use interface where relevant information is easily accessible.
The best thing with Elastic is that it scales in many ways. Not only can one add more data and keep getting sub-second response times, but one can also do so much more with the data. Even if THL is currently mostly using the log data for ensuring things run smoothly, the insights one can gain from the visualizations and the data is so fantastic that discussions are ongoing about using the same system for even more – and also visualizing business data.