> ## Content Index
> Fetch the complete content index at: https://www.autodidacts.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# Piping Hot: how hard can it be to have a realtime, unbuffered,  streaming, non-newline-delimited regex filter pipeline in Bash?
- URL: https://www.autodidacts.io/realtime-streaming-regex-filters-in-bash-pipeline-solve-sed-buffering-newline-problems-with-perl/
- Published: 2026-11-17T22:44:17.000Z
- Updated: 2026-11-17T23:01:03.000Z
- Description: 1,600 words hard
- Author: Curiositry
- Tags: 100DaysToOffload, Bash, Perl, Sed, Awk, Unix

I have spent hours on this problem more than once. I hope this saves someone from a bad afternoon.

### The problem

We are used to piping things to `grep` or `sed`, and getting output.

But, sometimes, the source isn't a list of lines. Sometimes, the text you want to run a regex replace on is a *stream*. Two real-life examples:

1. A stream of dictated text
2. Streamed output from Llama.cpp server

Here, we run into two issues:

1. Many coreutils buffer input and/or output
2. Many of the Unix coreutils operate on lines by default. Sed is the most obvious example.

### Test case

Here's an example that will let us test our solutions, and prove that most of them don't work.

Create a file:

```bash
touch /tmp/realtimestream
```

Now, tail it:

```bash
tail -f /tmp/realtimestream
```

Now, in a new terminal, append something to it, without a newline:

```
echo -n "Hello, tail. How are you?" >> /tmp/realtimestream

```

... and watch it arrive in terminal #2, as intended, the moment you hit enter.

Now, open another terminal, and try to filter the output of tail in realtime:

```bash
tail -f /tmp/realtimestream |  sed 's/([,;:.?!])/\1\n/g'

```

... and watch nothing come out when you run the echo command.

This is, like, a really dumb problem to have. We are trying to replace punctuation with newlines, but sed itself requires a newline before it processes the input, and there *aren't* any*.* And sed is strictly linewise: it doesn't let you set the delimiter.

### Solutions people suggest that don't work

Yes, pipelines are buffered, often. That can be part of the problem, but the main problem is that you can't use linewise tools on streaming input.

You will see many mentions of `stdbuf -o0 command`. It won't help, because that's not the problem. But once you fix the newlines problem, buffering might be your new problem. You'll also likely encounter the more comprehensive `stdbuf -i0 -o0 -e0 command` (that's setting the stdin, stdout, and stderr buffers to 0, rather than just stdout like the first example). As far as I can tell, you can have a space between the flag and the value, or not, so you might also see people write `stdbuf -i 0 -o 0 -e 0 command`.

`stdbuf` works for things that use LD\_PRELOAD, which isn't everything. If that doesn't work you can try `unbuffer command` (from the expect package), which is easier to remember, but may have side effects.

`sed -u` or `sed --unbuffered` doesn't help either, because it still needs newlines. 

`tr` doesn't need newlines. It operates on characters. So you can do `tr ',.:?!;' '\n\n\n\n\n\n'`, but it's not what we want, because it swallows the characters it matches against, replacing them, and doesn't support regex.

(For more, read [Julia Evans's fantastic post on buffering](https://jvns.ca/blog/2024/11/29/why-pipes-get-stuck-buffering/?ref=autodidacts.io).)

But, solving buffering still doesn't solve our problem.

### The one that got away

I thought this one was going to work. I've never grokked awk, and have only used snippets other people write. AI pointed me to [Record Splitting with Gawk](https://www.gnu.org/software/gawk/manual/html%5Fnode/gawk-split-records.html?ref=autodidacts.io#Record-Splitting-with-gawk), and I adapted the example, by trial and error, to this:

```bash
gawk 'BEGIN { RS = "([,.;:!?—])" } { print $0RT;} '
```

This *looks* like it's going to work. However, if you try it, you will find that though it outputs the first two split lines correctly, it swallows the last line, and saves it and outputs it at the *next* echo. So the first time you get:

```
Hello,
tail.
```

... and each subsequent echo:

```
How are you?
Hello,
tail.
```

This is, I gather, because there's never a terminator, so gawk remains hopeful/uncertain about whether more input is coming that might match the regex. No amount of `system("")` `fflush("")`ing will save you. (Believe me, I flushed everything: even paper towels.)

### Dinosaur riding 101

I have never used Perl much. I thought Perl was dead! It turns out, it's not: it just *looks* that way. (I jest.)

This approach was first recommended by AI. Then I threw out what the AI gave me and started from scratch with the sources it had pointed me to, so I would understand it better.

Here's the final snippet:

```bash
tail -f /tmp/realtimestream  | perl -pe '$| = 1; $/ = \1; s/([,;:.?!])/\1\n\n/g' | sed 's|are you|art thou|g'

```

Let's break it down:

We call perl with the `-pe` flags. This basically lets us write a script as an argument, and automatically prints the results. I found an explanation of flags in paxdiablo's answer [here](https://stackoverflow.com/questions/6302025/perl-flags-pe-pi-p-w-d-i-t?ref=autodidacts.io) ("*`-p`: Places a printing loop around your command so that it acts on each line of standard input. Used mostly so Perl can beat the pants off Awk in terms of power AND simplicity :-)*"). I think of -e as "eval", though I don't know if that's what it means.

AI recommended a BEGIN block. I'm not sure why, but it doesn't seem to be necessary, so I skipped it. See Eric Strom's answer [here](https://stackoverflow.com/questions/3998619/what-is-the-role-of-the-begin-block-in-perl?ref=autodidacts.io) if you want to know what BEGIN means. I recognized this from awk; I didn't know it was a Perl thing. I assumed that Awk got it from Perl, but no: Awk pre-dates Perl (more things I didn't know!), and according to [an eight year old Reddit comment by a deleted user](https://www.reddit.com/r/unix/comments/9gurzc/comment/e68spqf/?ref=autodidacts.io), Perl copied BEGIN from Awk. *And*, Awk is a full blown programming language (I always thought of it as one of the coreutils, not a programming language; but [maybe Kernighan did also](https://news.ycombinator.com/item?id=14647022&ref=autodidacts.io#:~:text=Brian%20Kernighan%20himself%2C%20in%20this%202015%20talk%20%5B1%5D%20on%20language%20design%2C%20states%20that%20Awk%20was%20primarily%20intended%20for%20one-liner%20usage%20%28he%20mentions%20this%20at%2020%3A43%29.)?). There's just so much I don't know, and that I would have never learned if I hadn't gone to the sources.

If there's one thing we've learned, it's that most of the coreutils buffer input/output, and most of them have a flag to disable it.

`$| = 1` is how you tell Perl not to buffer stdio, according to *Automating System Administration with Perl, 2nd Edition* by David N. Blank-Edelman (O'Reilly, 2009) — which AI provided me with a free PDF of, hosted on a Chinese website — and also according to [perlvar's section on output autoflush](https://perldoc.perl.org/perlvar?ref=autodidacts.io#$OUTPUT%5FAUTOFLUSH) ("*Mnemonic: when you want your pipes to be piping hot*."), and a StackOverflow thread that I lost.

Then, we tell Perl to read input one character at a time, using the `$/ = \1` "special construct" described [here](https://stackoverflow.com/questions/8676026/how-can-i-have-perl-take-input-from-stdin-one-character-at-a-time?ref=autodidacts.io), which is also discussed in [this perlmonks thread](https://perlmonks.org/?node%5Fid=302692&ref=autodidacts.io), and in the [perlvars section on the input record separator](https://perldoc.perl.org/perlvar?ref=autodidacts.io#$INPUT%5FRECORD%5FSEPARATOR). Apparently this is really inefficient and there are better ways to do it, but for my use case, it doesn't matter, and I chose the simple solution that I could understand.

 *(Fun fact: make sure the slashes are going opposite directions, like a pyramid. When I typoed that first slash (*`$\ = \1`*), it worked, but kept spitting out* `SCALAR(0x637398f65be0)` in with the output, which had me very confused until I noticed my mistake.*)*

Another gotcha: if we are reading one character at a time, what about unicode? Can we regex an em-dash? I thought this wouldn't work. But it *partly* works. It appears to match the em-dash. However, the capture group replacement doesn't work with em-dashes, so they can be passed through without getting matched, or they can be matched, but they get swallowed. However, you can use a literal em-dash in the replacement, so you could match them in a separate Perl snippet, and replace them with "—\\n". Unwieldy, but it would work! There's likely a better solution that I am not motivated enough to find.

`s/search/replacement/g` Finally, we get to the thing we have been trying to do, in order to do the thing we have been trying to do. And the syntax is just like `sed`, so I immediately felt at home. (Who got it from whom?!) I have `s/([,;:.?!]{1,4})/\1\n/g`, which appends a newline after punctuation (excepting em-dashes, ellipses, and actual interrobangs). This means sed gets what it wants: newline delimited data.

Finally, we can filter our stream!

`sed 's|are you|art thou|g'` which `s`ubstitutes search with replacement `g`lobally (ie, not just first match).

Now, try it. Run the whole thing:

```bash
tail -f /tmp/realtimestream  | perl -pe '$| = 1; $/ = \1; s/([,;:.?!])/\1\n\n/g' | sed 's|are you|art thou|g'

```

And then, in another terminal:

```bash
echo -n "Hello, tail. How are you?" >> /tmp/realtimestream

```

It works!

We are done. We are *so*. *done.*

---

**Appendix**

There's a lot of choice humour to be found in the primary sources on this stuff, which you might expect to be dry and academic. From the Wikipedia page on Perl:

> It has been nicknamed "the Swiss Army chainsaw of scripting languages" because of its flexibility and power.[\[21\]](https://en.wikipedia.org/wiki/Perl?ref=autodidacts.io#cite%5Fnote-21) In 1998, it was also referred to as the "[duct tape](https://en.wikipedia.org/wiki/Duct%5Ftape?ref=autodidacts.io) that holds the [Internet](https://en.wikipedia.org/wiki/Internet?ref=autodidacts.io) together", in reference to both its ubiquitous use as a [glue language](https://en.wikipedia.org/wiki/Glue%5Flanguage?ref=autodidacts.io) and its perceived inelegance.[\[22\]](https://en.wikipedia.org/wiki/Perl?ref=autodidacts.io#cite%5Fnote-leonard98-22)

And, perusing *The GNU Awk User's Guide* [¶ 9.1.5 Input/Output Functions](https://www.gnu.org/software/gawk/manual/html%5Fnode/I%5F002fO-Functions.html?ref=autodidacts.io#Input%5F002fOutput-Functionshttps://www.gnu.org/software/gawk/manual/html%5Fnode/I%5F002fO-Functions.html#Input%5F002fOutput-Functions), we find this paragraph:

> In the past, `awk`’s `system()` function simply returned the exit status value divided by 256\. In the normal case this gives the exit status but in the case of death-by-signal it yields a fractional floating-point value.[56](https://www.gnu.org/software/gawk/manual/html%5Fnode/I%5F002fO-Functions.html?ref=autodidacts.io#FOOT56) POSIX states that `awk`’s `system()` should return the full 16-bit value.

And, way down yonder, in footnote 56:

> In private correspondence, Dr. Kernighan has indicated to me that the way this was done was probably a mistake.