Piping Hot: how hard can it be to have a realtime, unbuffered, streaming, non-newline-delimited regex filter pipeline in Bash?
1,600 words hard
I have spent hours on this problem more than once. I hope this saves someone from a bad afternoon.
The problem
We are used to piping things to grep or sed, and getting output.
But, sometimes, the source isn't a list of lines. Sometimes, the text you want to run a regex replace on is a stream. Two real-life examples:
- A stream of dictated text
- Streamed output from Llama.cpp server
Here, we run into two issues:
- Many coreutils buffer input and/or output
- Many of the Unix coreutils operate on lines by default. Sed is the most obvious example.
Test case
Here's an example that will let us test our solutions, and prove that most of them don't work.
Create a file:
touch /tmp/realtimestreamNow, tail it:
tail -f /tmp/realtimestreamNow, in a new terminal, append something to it, without a newline:
echo -n "Hello, tail. How are you?" >> /tmp/realtimestream
... and watch it arrive in terminal #2, as intended, the moment you hit enter.
Now, open another terminal, and try to filter the output of tail in realtime:
tail -f /tmp/realtimestream | sed 's/([,;:.?!])/\1\n/g'
... and watch nothing come out when you run the echo command.
This is, like, a really dumb problem to have. We are trying to replace punctuation with newlines, but sed itself requires a newline before it processes the input, and there aren't any. And sed is strictly linewise: it doesn't let you set the delimiter.
Solutions people suggest that don't work
Yes, pipelines are buffered, often. That can be part of the problem, but the main problem is that you can't use linewise tools on streaming input.
You will see many mentions of stdbuf -o0 command. It won't help, because that's not the problem. But once you fix the newlines problem, buffering might be your new problem. You'll also likely encounter the more comprehensive stdbuf -i0 -o0 -e0 command (that's setting the stdin, stdout, and stderr buffers to 0, rather than just stdout like the first example). As far as I can tell, you can have a space between the flag and the value, or not, so you might also see people write stdbuf -i 0 -o 0 -e 0 command.
stdbuf works for things that use LD_PRELOAD, which isn't everything. If that doesn't work you can try unbuffer command (from the expect package), which is easier to remember, but may have side effects.
sed -u or sed --unbuffered doesn't help either, because it still needs newlines.
tr doesn't need newlines. It operates on characters. So you can do tr ',.:?!;' '\n\n\n\n\n\n', but it's not what we want, because it swallows the characters it matches against, replacing them, and doesn't support regex.
(For more, read Julia Evans's fantastic post on buffering.)
But, solving buffering still doesn't solve our problem.
The one that got away
I thought this one was going to work. I've never grokked awk, and have only used snippets other people write. AI pointed me to Record Splitting with Gawk, and I adapted the example, by trial and error, to this:
gawk 'BEGIN { RS = "([,.;:!?—])" } { print $0RT;} 'This looks like it's going to work. However, if you try it, you will find that though it outputs the first two split lines correctly, it swallows the last line, and saves it and outputs it at the next echo. So the first time you get:
Hello,
tail.... and each subsequent echo:
How are you?
Hello,
tail.This is, I gather, because there's never a terminator, so gawk remains hopeful/uncertain about whether more input is coming that might match the regex. No amount of system("") fflush("")ing will save you. (Believe me, I flushed everything: even paper towels.)
Dinosaur riding 101
I have never used Perl much. I thought Perl was dead! It turns out, it's not: it just looks that way. (I jest.)
This approach was first recommended by AI. Then I threw out what the AI gave me and started from scratch with the sources it had pointed me to, so I would understand it better.
Here's the final snippet:
tail -f /tmp/realtimestream | perl -pe '$| = 1; $/ = \1; s/([,;:.?!])/\1\n\n/g' | sed 's|are you|art thou|g'
Let's break it down:
We call perl with the -pe flags. This basically lets us write a script as an argument, and automatically prints the results. I found an explanation of flags in paxdiablo's answer here ("-p: Places a printing loop around your command so that it acts on each line of standard input. Used mostly so Perl can beat the pants off Awk in terms of power AND simplicity :-)"). I think of -e as "eval", though I don't know if that's what it means.
AI recommended a BEGIN block. I'm not sure why, but it doesn't seem to be necessary, so I skipped it. See Eric Strom's answer here if you want to know what BEGIN means. I recognized this from awk; I didn't know it was a Perl thing. I assumed that Awk got it from Perl, but no: Awk pre-dates Perl (more things I didn't know!), and according to an eight year old Reddit comment by a deleted user, Perl copied BEGIN from Awk. And, Awk is a full blown programming language (I always thought of it as one of the coreutils, not a programming language; but maybe Kernighan did also?). There's just so much I don't know, and that I would have never learned if I hadn't gone to the sources.
If there's one thing we've learned, it's that most of the coreutils buffer input/output, and most of them have a flag to disable it.
$| = 1 is how you tell Perl not to buffer stdio, according to Automating System Administration with Perl, 2nd Edition by David N. Blank-Edelman (O'Reilly, 2009) — which AI provided me with a free PDF of, hosted on a Chinese website — and also according to perlvar's section on output autoflush ("Mnemonic: when you want your pipes to be piping hot."), and a StackOverflow thread that I lost.
Then, we tell Perl to read input one character at a time, using the $/ = \1 "special construct" described here, which is also discussed in this perlmonks thread, and in the perlvars section on the input record separator. Apparently this is really inefficient and there are better ways to do it, but for my use case, it doesn't matter, and I chose the simple solution that I could understand.
(Fun fact: make sure the slashes are going opposite directions, like a pyramid. When I typoed that first slash ($\ = \1), it worked, but kept spitting out SCALAR(0x637398f65be0) in with the output, which had me very confused until I noticed my mistake.)
Another gotcha: if we are reading one character at a time, what about unicode? Can we regex an em-dash? I thought this wouldn't work. But it partly works. It appears to match the em-dash. However, the capture group replacement doesn't work with em-dashes, so they can be passed through without getting matched, or they can be matched, but they get swallowed. However, you can use a literal em-dash in the replacement, so you could match them in a separate Perl snippet, and replace them with "—\n". Unwieldy, but it would work! There's likely a better solution that I am not motivated enough to find.
s/search/replacement/g Finally, we get to the thing we have been trying to do, in order to do the thing we have been trying to do. And the syntax is just like sed, so I immediately felt at home. (Who got it from whom?!) I have s/([,;:.?!]{1,4})/\1\n/g, which appends a newline after punctuation (excepting em-dashes, ellipses, and actual interrobangs). This means sed gets what it wants: newline delimited data.
Finally, we can filter our stream!
sed 's|are you|art thou|g' which substitutes search with replacement globally (ie, not just first match).
Now, try it. Run the whole thing:
tail -f /tmp/realtimestream | perl -pe '$| = 1; $/ = \1; s/([,;:.?!])/\1\n\n/g' | sed 's|are you|art thou|g'
And then, in another terminal:
echo -n "Hello, tail. How are you?" >> /tmp/realtimestream
It works!
We are done. We are so. done.
Appendix
There's a lot of choice humour to be found in the primary sources on this stuff, which you might expect to be dry and academic. From the Wikipedia page on Perl:
It has been nicknamed "the Swiss Army chainsaw of scripting languages" because of its flexibility and power.[21] In 1998, it was also referred to as the "duct tape that holds the Internet together", in reference to both its ubiquitous use as a glue language and its perceived inelegance.[22]
And, perusing The GNU Awk User's Guide ¶ 9.1.5 Input/Output Functions, we find this paragraph:
In the past,awk’ssystem()function simply returned the exit status value divided by 256. In the normal case this gives the exit status but in the case of death-by-signal it yields a fractional floating-point value.56 POSIX states thatawk’ssystem()should return the full 16-bit value.
And, way down yonder, in footnote 56:
In private correspondence, Dr. Kernighan has indicated to me that the way this was done was probably a mistake.