atop logs are not limited in size
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	zsdc
	Aug 24 2021, 5:15 PM

Description

Atop stores accounting logs in two places: /run/atop/ and /var/log/atop/. By default, each day data from the first one moves to the second one. Also, data in the /var/log/atop/ rotating to keep only 28 last days (default value that we did not change).
The problem is that in rare cases files may be big enough to take more place than expected and we have no options to predetermine maximum size for both places where atop stores data.

To avoid potential problems with low memory and low storage deployments, we need to allow controlling a maximum size of both current and historical log files. It seems that two additional logrotate rules may help here.

Details

Difficulty level: Normal (likely a few hours)
Version: 1.4-rolling-202108240117, 1.3-beta-202108230342
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Behavior change
Issue type: Bug (incorrect behavior)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	BUG	zsdc	T3774 atop logs are not limited in size
		Resolved	FEATURE REQUEST	zsdc	T2613 add logrotate options for syslog

Event Timeline

zsdc created this task.Aug 24 2021, 5:15 PM

How about CLI set system syslog atop file 5
That means save the latest 5 files.

Also, I don't find anything in /run/atop/

vyos@r1-roll# ls -la /run/atop
ls: cannot access '/run/atop': No such file or directory
[edit]
vyos@r1-roll#

let's do it but not in syslog section

syncer triaged this task as Normal priority.Oct 17 2021, 1:41 PM

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0-epa3); removed VyOS 1.3 Equuleus.

@zsdc Provide please options to solve it.

After some investigation, we figured out several ways how to solve or at least mitigate the problem. From my point of view, the optimal for both developers and customers is the next one.

Depending on the package version, atop uses crontab or systemd timer for rotating logs and restarting the service (should be restarted for rotation).
Logrotate in its turn, use its own logic that can be controlled by us/customers more carefully. The problem here is that these two rotations are almost incompatible, which leads to multiple problems and unreasonable complex logic to combine them together.

Therefore, I think that we should disable all the services, cron items, timers that come with the atop package (because they are all tightly connected), and then do the next:

Create our own atop systemd service or overwrite settings in the default one. This allows us to control exec options as we want.
Add to CLI setting that will control: count of log files to keep, the maximum size of each file, the maximum days to keep log files.
Based on these CLI options, create a logrotate config file that will monitor the atop file and rotate it if necessary. Restarting service during rotation will also be done by logrotate (currently this is done by atop scripts).

With this, we can set default size and give the ability to change it if necessary.

Nuances that we need to know:

logrotate timer should be changed to more often (currently it runs daily), otherwise, log files may grow up too fast and take too much space before rotation.
the individual size of log (also as total) can be bigger than a configured value because logrotate will trigger the condition only when size will overcome it. Unfortunately, there is no way to change this, regardless of rotation strategy or tools.

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus (1.3.0-epa3).Nov 1 2021, 10:11 PM

pasik added a subscriber: pasik.Nov 2 2021, 8:59 AM