<- Back
Comments (93)
- otterleySomething must have happened along the way, because this was not the original design intent of the database (emphasis mine):"""The native journal file format is inspired by classic log files as well as git repositories. It is designed in a way that log data is only attached at the end (in order to ensure robustness and atomicity with mmap()-based access), with some meta data changes in the header to reference the new additions. The fields, an entry consists off, are stored as individual objects in the journal file, which are then referenced by all entries, which need them. This saves substantial disk space since journal entries are usually highly repetitive (think: every local message will include the same _HOSTNAME= and _MACHINE_ID= field). Data fields are compressed in order to save disk space. The net effect is that even though substantially more meta data is logged by the journal than by classic syslog the disk footprint does not immediately reflect that."""See https://docs.google.com/document/u/0/d/1IC9yOXj7j6cdLLxWEBAG...
- jck86The cherry on the cake is that you practically cannot filter journald. The only option is limiting by severity (e.g. errors and higher) or switch to non persistent journald storage and forward to rsyslog and filter there.Am a bit vague on the details but sometimes a driver goes bezerk and starts logging many times per second, e.g. a bug in amdgpu after resume from suspend. Took a while to get that filtered which luckily was only possible because it were kernel messages (dmesg), but for a while I had to disae persistent kernel logging which is dat from ideal.I get that for certain core parts simplicity is more important than features. But journald is just too basic to enable persistent storage but I also don't want to switch it off.
- barrkeljournald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
- 0x_rsjournald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:[Service]LogFilterPatterns=~I am a completely useless log entryLogFilterPatterns=~I am another useless log entryBut it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.
- smartmicI recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedomWill try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
- pudgywalshHow do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?The first thing I do on a Linux system is install a proper syslog daemon.
- ValdikSS
- amlutoOoh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.
- d3Xt3rOkay, so how do I disable journald and switch to something else, without getting rid of systemd completely?
- skulloneThe "gift" of systemd never stops. It's like an STD, just spreading rot across everything it touches
- sam_lowry_Cool to see @ValdikSS here as well. The guy never sleeps or he is AI in disguise ;-)
- pengaruI'm probably the main person responsible for making journald usable at all.But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon.Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog.My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size.Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written.The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem.
- hedoraSomeone should implement a new operating system that can efficiently handle text processing.It could have some simple tools that let you generate reports, display them on screen, and compose tools for that sort of thing in a natural way.We could call it UNIX.
- lokarFor 99% of installs the basic assumption that local logging (with local reading) is the primary mode is just wrong.
- sidewndr46years ago, I set Storage=volatile on almost all the journalD configurations I have. This largely solved this kind of problem.
- otterleyThis issue report feels like it ought to be accompanied by a fix. If you think you can do better than journald's existing format, propose a new one with tests to prove it. GenAI makes this much easier than it used to be.
- tryauuumhello ValdikSS! nice to see you alive
- greatgibSystemd things being horse shit as usual because it was vibecoded even before LLM existed. And there are still people that said that systemd and tools are awesome because they never encountered any of the countless ridicule bugs.
- mono442journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
- raszOh how I love totally predictable poetterings reply to previous bug report that got closed because "measuring it wrong" and "this is not a support forum".
- quotemstrSystemd should just use DuckDB. It's perfect for this job."But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.Seems like there are a lot of mistaken impressions about DB storage engines out there.