I worked at a company that, for security reasons, did not give developers direct access to the logs for the applications they deployed. Instead, developers had access to a 3rd party app that conglomerated all the logs of all the applications of all of the environments. If you can see logs for your actions as soon as they occur, then there’s nothing wrong with this. The problem was that this 3rd party app would become a bottleneck that all logs had to go through. As a result, there could be up to a 30 minute delay for some logs.
This creates an unreasonably long feedback loop for developers who are trying to debug issues. Imagine you think you’ve fixed an issue so you test it and then you have to wait 30 minutes to see if it worked. What if it didn’t? Well, there goes a whole hour down the drain.
There are many ways to solve this problem but they all result in the same outcome: Developers need instantaneous feedback from logs. Even a 5 second delay can be troubling. Because when a production server generates a lot of noise, it can be difficult to figure out what logs you’re generating and what everyone else is generating. This is difficult even when there’s a zero second delay. Any amount of delay exacerbates the issue.
If you need a 3rd party app for security purposes, make sure it doesn’t delay the ability to view logs. This is like taking away a carpenter’s hammer. These are tools developers need. Instead, find a log viewer that has no delay. If the log viewer has so much data going through it that it has a delay, then figure out a way to cluster these viewers onto multiple servers.
For some, it may be more convenient to have the logs in one place. But for developers, it is not only inconvenient, but unproductive. Those people that would prefer to have the logs in one place should have their own dedicated, laggy server that they can use at their leisure. Let the developers have a blazing fast one with instantaneous feedback.