We all have data that we need to store, and then find. Regardless of type, data tends to build up. Eventually, we need some system for organizing it into sensible categories.

It turns out, this problem is harder than it seems.

In this article, I'm going to be talking about organizing digital files. However, most of the problems (and some of the solutions) also apply to paper files, spreadsheet tables, databases — and even physical objects.

Let's dig in.

The Naïve Categorization System

Starting out, most of us come up with a categorization system that makes intuitive sense, following the general guideline a place for everything and everything in its place.

Everyone's mind works differently. The structure of people’s categorization systems tend to reflect the way their mind works. Some people have hundreds of files piled in visual heaps on the digital (or physical) desktop, and it works for them. But most of us have some kind of nested folder structure, and most people I know organize it by type or project.

Here's an example:

writing/
    fiction/
        in progress/
        draft/
        submitted/
        published/
    non-fiction/
    poetry/
        song lyrics/
    blog/
        post title/
reading/
photos/
learning/
music/
    recordings/
    practice and learning materials/
    original music/
    traditional songs and covers/

Obviously this is incomplete, but you get the idea. Organize things by context, in a way that makes sense, without overthinking it.

But you're probably already overthinking it, and if so, you have likely noticed several serious taxonomic flaws with the above directory structure!

The Problems: Examples

Exhibit A: The song lyrics directory is a subdirectory of writing. But song lyrics also belong in the music directory.

Exhibit B: Some of the photos in the photos directory are used in the blog that's in the writing directory.

I can see five ways of addressing Exhibit B:

1. Move the photos that are used on the blog in the relevant post folder in the blog directory — and lose the convenience of having all photos in one place.

2. Make a blog subdirectory in photos with all the photos that are used on the blog — and then never know exactly which photos go with which post.

3. Copy the photos that are used by the blog into the relevant blog subdirectory, leaving the canonical original copy with the original filename in the photos directory — and then have duplicates of those photos on the filesystem.

4. Get rid of the photos directory altogether, and put all photos in the folder of the project they belong to.

5. Give up and bury my head in the sand.

All the options are kind of horrible. But the last one seems promising.

The Naïve Solution: Project-Based Taxonomies

I’ve had friends recommend — and I have started to move to — a project based categorization system. It solves many of the problems. For example, we don't have the problem of where to put speculative poetry (fiction, or poetry?), or any of the other problems that the above brittle hierarchy exhibits. Instead, we have a folder for each project (song, story, essay, etc.) that everything related to that project goes in: text, photos, reference material, everything.

Most often, the context we are working in is single-project, so this works well. I sit down to work on project X, not to work on “Lists and Spreadsheets” (another directory; not kidding). I have everything I need to work on that project, right at my fingertips.

But we still have two problems: duplication of files that are used in multiple projects, and loss of convenient access to files by type.

The Problems, in Abstract

1. The re-use problem. Data is used by multiple projects.

   • Does a book review belong in reading, or writing?

2. The interface problem. Sometimes we want to view data by type, not by project.

   • That photos folder, even though it causes all kinds of problems, is there for a reason: sometimes we want to look at all the photos.

3. The mixed-quality problem. For every project, there is data that is invaluable and crucial, as well as a bunch of cruft created along the way. Project-based taxonomies tend to mix this all together in a way that is unpleasant.

   • Sometimes we want to exclude low-quality or third-party data from back-ups, due to limited space (or time).

4. The project-boundary problem. The boundaries of a “project” are fuzzy.

   • I have an algebra rules website. I'm working on an algebra rules poster. I abandoned an algebra rules course. Are these three projects, or one?

Symlinks to the Rescue?

We can solve two problems at once by using symbolic links. We can have our avocado toast and eat it too, by having type folders and project folders!

photos
  repellent-author-portrait.jpg

mangy-blogpost
  symlink-to-repellent-author-portrait.jpg
navelgazing-short-story
  another-symlink-to-same-repellent-author-portrait.jpg

Now, we have projects that contain everything we need, and we have a convenient type-based interface for all photos. Plus, we only store one copy of the file on disk.

Have we solved the hard problems of taxonomies yet?

Uh... not quite.

1. Symlinks are brittle. As soon as you re-organize the photos folder, or rename something further up the tree, all the symlinks to that original .jpg break. It sucks. Spreadsheets update references when data is moved; why can't we have nice things like that on the filesystem?

2. Having a bajillion symlinks to one file is gross. I don't know what it is exactly, but there's something ... messy about this approach. Symlinks aren’t technically duplicate files, but they feel like they are.

Tags to the Rescue!

Directories are hierarchical, and life is not. The best solution I’ve come up with is to give up on directory structures and symlinks, and use tags for all organization.

Now, we can tag that author photo with project:storytitle, project:essaytitle, and type:photo. This means:

1. Zero file duplication (not even symlinks!)

2. Multiple convenient overlapping “interfaces” — just view all files tagged project:storytitle, or type:photo (you could call it project:photos)

3. Freedom from the tyranny of rigid directory structures. Throw all your files in one giant potpourri folder — or keep your existing broken directory structure as is. Tag-based taxonomies enhance your existing organization system, rather than replacing it. (You can use your existing directory structure to automatically apply project and type tags to your files.)

File Tagging in the Real World

We have a promising theoretical solution to The Hard Problem of Taxonomy. But the software to implement it is still in diapers.

Metadata Storage

There are several places tags for a file can be stored. None of them are a panacea.

1. Extended file attributes (called “alternate data streams” on Windows, and “named forks” on Mac) are a filesystem-level way of storing metadata. They are ‘attached’ to the file in a sense, and travel with it where extended attributes are supported, but they aren't embedded in the file itself.

   • Pros: works with all filetypes, all major operating systems support them in some form, they travel with the file when it is moved or renamed.
   • Cons: not many tagging tools support them, poor operating system level integration on Linux and Windows, different operating systems’ implementations are largely non-interoperable, can only store ~4kb of metadata on some filesystems, poor portability.

2. Sidecar files. These are typically plaintext or XML files that are hidden by default, most often stored 1:1 alongside the single file the metadata sidecar file refers to.

   • Pros: portable, simple, no size or characterset limits for metadata contained in them.
   • Cons: multiple competing non-interoperable formats for tag sidecar files, creates clutter, hidden files can get left behind, can easily become unlinked if the file referred to is moved or renamed.

3. File-specific embedded metadata such as EXIF data, ID3 tags, Vorbis comments, IPTC keywords, et cetera.

   • Pros: travels with the file, generally well-supported by operating systems and other software, industry standard for media files.
   • Cons: not all filetypes have a place for it, different file formats have different fields.

4. Symbolic links. I'm not sure if it’s ever done, but technically you could have tag directories, and then symlink files into them? It would probably be a putid approach. More often, files are tagged, and the file-tag relations are stored in a database, and symlinks are used as described below.

   • Pros: simple.
   • Cons: very primitive approach, breaks when files are moved.

5. In the filename. This is such a bad idea I don't even want to talk about it. However, who hasn’t tried it?

   • Pros: simple, portable, widely supported.
   • Cons: breaks your file naming system, makes your filenames horrible, limits tag length, limits characters that can be used in tags, may break spectacularly when moved to an OS with tighter constraints on allowed filenames.

6. A database. Rarely used on its own, though it could be. Generally, a database is used along with a virtual filesystem, with symlinks to the actual tagged files. A database is also generally required for any type of metadata that doesn't automatically travel with the file (such as sidecar files). A database is also generally needed to index tags stored in embedded metadata, or extended attributes. Storing tags only in a database seems like a bad idea, but pretty much any functional file tagging system is going to need some kind of database (often SQLite).

   • Pros: you kinda need one.
   • Cons: Fragile, not necessarily portable, needs some kind of filesystem watcher that keeps it updated.

Operating System Support for File Tagging

Here’s the state of things:

1. MacOS has built in, filesystem-level tagging via extended file attributes. Tags are also stored in .DS_Store (a sidecar file that's already used for other things) as a fallback. These tags have first class-support in Finder and Spotlight.

2. Windows supports tagging some types of files using IPTC keywords. The file manager has support for finding things via tags, but not for managing tags or tagging files in bulk. Windows also has its own version of extended attributes (“alternate data streams”), but nothing much uses them. They’re used to to store the url of files downloaded from the internet, so that Windows can display a warning when the user tries to run them.

3. Linux is still in the stone age when it comes to tagging. Common Linux filesystems (including ext4 & ZFS) support extended attributes, but I'm not aware of any Linux distro or file manager that includes tagging features based on them (or embedded metadata, for that matter).

Third-Party Tagging Software for Linux

Third-party tagging software for Linux includes command-line tools
TMSU, SuperTag, and Tagsistant, and a cross-platform GUI app called TagSpaces (warning: embeds tags in filename by default!). TMSU seems to be the most robust and recently updated contender. However, most of these are built on a database + virtual filesystem + symlinks, and inherit their problems. Ick!

I don’t know why none of the options use extended attributes. They could use the setfattr & getfattr commands from the attr package. Likely it has to do with the fact that other apps (including core utilities like cp, without a specific flag) tend to destroy data stored in extended attributes, they aren't preserved by .zip compression (tar with --xattrs flag preserves them), and not all filesystems (notably, FAT32) support them in the first place. For more on extended attributes (which seem like the most promising option) see Extended attributes: the good, the not so good, the bad.

It might be possible to keep symlinks from going stale with a file manager plugin that updates them when files are moved or renamed, or an indexer — but I’ve asked, and haven’t heard of any existing solution.

The more I look into TMSU, though, the more promising it looks. It doesn’t keep symlinks updated automatically, but it has a repair function that will reattach tags to files that have been moved or modified — as long they have not been both moved and modified! TMSU also offers commands for moving/renaming/deleting files while keeping tags fresh, and someone is even working on a Nautilus extension. Best of all, based on issues #10, #86, and #160, the developer seems open to the idea of supporting extended attributes, and has added them to the v0.8.0 milestone.

(In)conclusion

While we wait for better software (or go build it!), we can go ahead and use something like TMSU to tag files — and keep our existing (imperfect, but more reliable) directory structure as a fall-back.

What are the limitations of tag-based organization? I have only just started moving my data to tag-based systems, so I can't say how well it works at scale. Tags have worked well in the few areas where I have been using them. From my reading, a common complaint is that tags get out of hand, and end up with duplicates and general inconsistency — requiring labour-intensive maintenance.

When I mentioned the problem of inconsistent tags to my brother — who has been using embedded-metadata tags for decades as a professional photographer — he patted me on the back and said, “Young man, you are at the start of a very long journey.”

Elon Musk is planning to colonize Mars. If he succeeds, it could be one of the most important changes in human history. And, based on the surprising success of his previous ventures — including Zip2, PayPal, Tesla Motors, and SolarCity — it might be foolish to dismiss Musk’s plan.

There has been a lot of hype about Mars, beginning with the Schiaparelli’s illusory observation of Mars “canali” (mistranslated from Italian to the English as “canals”^[1]) in 1877 and culminating in the media hyperventilation over Musk announcing, at the 67th International Astronautical Congress in Guadalajara, Mexico, his plan to send cargo to Mars in three years and colonists in ten.^[2]

The prospect of my generation reaching Mars has fired up the popular imagination about interplanetary spaceflight in a way not seen since the Sputnik and Apollo missions six decades ago.

Everyone’s excited about rockets to Mars, and each SpaceX launch brings that dream closer to reality^[3]. Musk and others are putting a lot of money and brainpower on the technical problem of getting people to Mars. Less sensational topics, such as surviving on Mars, receive less attention — but plenty of money and serious thought, because there’s no way to get around them.^[4]

But there’s another important question which isn’t getting much attention:

Who will own Mars, and how will it be governed?

Does Mars belong to the people who get there first? To the highest bidder? To all the people of Earth?

Does Mars belong to Earth, or does Mars belong to Mars? Does it belong to the Sun? To the Martian microbiome, if there is one? (What are the indigenous rights of microbes, I wonder?)

Who will be in charge of Mars once the colonists arrive? If Mars turns out to have valuable resources, who gets them? And if a Mars colony is to govern itself, what kind of government would it have?

The Mars colonization project is driven by the ultra rich. And those who want to stake their claim on Mars may rather the rest of us didn’t think too much about the little problem of who owns the planet next door, and why.

Compared to most tech entrepreneurs, or most billionaires, Elon Musk seems to be doing a lot of good — he’s been pioneering (and funding) electric transportation, clean energy, and more prudent approaches to AI research. Now, he wants to backup the human race. But others in the same club may not be so altruistic — and may put business interests, such as mining Martian resources, ahead of good governance.

When it comes to governance of a Martian colony, the first question is this: will Mars be controlled from Earth, or from Mars? Initially, running Mars from Mars will probably be impractical. There will be too few people, they’ll still be dependent on Earth, and they’ll have their hands full surviving.

Earth-based control of the Martian colony could run the gamut from first-come-first-served resource extraction, to some kind of bureaucratic transitional governing body dispensing resource and settlement rights, to a more carefully regulated “commons.”

But, immediately upon the establishment of a stable colony, the interests of the Martian settlers will begin to diverge from those of their terrestrial bosses, just as they have in Earth’s colonial past. To not put in a plan for transition to self-government would be to risk disgruntlement, and, eventually, revolt.

Yet, a self-governing planetary neighbour presents its own challenges. There are plenty of scenarios that could lead to a “rogue colony” of one kind or another that would be, if not a military threat in the short term, at least an annoying and potentially inconvenient political situation.

And then there’s the form of governance, regardless of where it’s controlled from. We have all the options that we’ve tried here on Earth (such as monarchy, theocracy, dictatorship, plutocracy, and representative democracy), few of which we’d want to repeat, as well as possibilities we haven’t yet imagined.

Let’s plot a few of the possibilities to get a better look at them:

In the lower left we have Mars as an international commons — neutral territory shared by all Earth’s nation-states. Antarctica is the closest analogue I’m aware of here on earth, and it’s not that close. (When we discovered it, we didn’t know it was loaded with natural resources, and I suspect this fact, along with the fact that it’s an unwelcoming hunk of ice, made it possible to put such peaceful and ecologically-responsible treaties in place. The same can’t be said for Mars.)

Musk has mentioned some sort of internet-facilitated direct democracy^[5]. A direct democracy could be a way to mitigate some of the weaknesses of human nature without altering human nature itself. Whoever goes to Mars will bring themselves with: their entire cultural inheritance (beliefs, prejudices, and so forth) will go along for the ride, and so will human nature. That’s the problem: wherever we go, there we are. But as far as politics goes — and it only goes so far — direct democracy may be as good an option as any.

We have a long history of bad governance, war, corruption, injustice, and inequality. We’ve run through the laundry list of possible political problems more than once. If we don’t re-think our approach to governance before we colonize Mars, we risk repeating the most sordid parts of our history here on Earth, and Mars inheriting our intellectual, moral, and political diseases.

Musk proposes Mars as a “backup” for humanity, but, despite the apparent prudence of the idea, a second planet full of humans also doubles the chance of planetary-scale catastrophe. These concerns may seem too far-fetched and too far-future to be worth worrying about, but I think they’re worth considering because the choices being made now will have major ramifications for a Martian future. We’re having plenty of trouble here on Earth. Interplanetary conflict — human or microbial — is the last thing we need^[6].

This is the fear-based, reactionary reason for figuring out Martian governance. There’s another profoundly hopeful reason: Mars could be an opportunity to get governance right. For once, we can do what it takes without bloodshed or revolution, because nothing needs to be overturned.

Maybe we need a new political system. Maybe the solution is a lie detector, an empathy test. Maybe it’s mandatory meditation. Maybe the solution is a new self-improvement webinar that would keep the less altruistic aspects of human nature in check. Maybe we need to build A.E Van Vogt’s Games Machine.

When the question first occurred to me, I figured I’d be able to come up with a solution. The more I thought, the more thorny the issue became: it is now clear, to my great disappointment, that I don’t know how to govern Mars. But I think it’s our collective responsibility to try to ensure, in whatever way we can, that Mars is governed well. We’re members of the human race, and the human race is trying to colonize Mars: a lot is at stake in how this project works out. Which is why the rest of us should do what we can, even if the closest we’ll get to governing Mars is thinking about it, talking about it — or writing an essay and casting it adrift in some internet backwater.

When it comes down to it, I think many of the problems we’ve had on Earth are due to human nature. But our societies reward some aspects of human nature, and ignore others. Maybe we can change ourselves from the outside in. Maybe we need to change the world from the inside out. Maybe we need to do both.

Thanks for reading. If you have a question, an essay, or a grammatical axe to grind, leave a comment below. If you can say it in under 140 characters, you can find me on twitter. Send me an email if you want to talk without (too much of) the world watching.

While shopping for a new Linux laptop, I attempted to use Google-style advanced search operators on eBay, without luck.

All references to such things seem to have been removed from their docs. But it turns out it’s still possible.

Here’s the syntax I've sleuthed out:

• AND operator: thing1 +thing2
• OR operator: (thing1,thing2)
• NOT operator: thing1 -thing2

Notice that, unlike Google, there is no whitespace between the AND/NOT operator and the following term.

Here’s an example which includes the syntax for AND, OR, NOT operators:

ryzen +(7,9) +(t14,z13,x13,z16,g14,g16,"zephyrus","framework laptop",g15,"tuxedo","starlabs") -3750H

So far, I haven’t managed to get grouping to work within an OR statement (ie, “([ryzen +(7,9)],"framework") laptop” or “((ryzen +(7,9)),"framework") laptop”).

While we’re on the topic, Craigslist also supports this, with a more intuitive syntax:

(t14 | z13 | x13 | z16 | g14 | g16 | "zephyrus" | "framework laptop" | g15 | "tuxedo" | "starlabs") ("ryzen 7" | "ryzen 9")

I do a lot of Ghost theme development, and have to swap which theme is running on my local install all the time. The Ghost admin interface takes a lot of clicking to get to the page where you can change themes. And I don’t like clicking.

Neither Google nor GPT could tell me how to change the Ghost theme from the command line, using the Ghost Admin API or Ghost CLI.

Fortunately, standard Unix tools can get the job done.

In preparation:

1. Make sure you don’t have a theme installed called “current”.
2. Create a symbolic link called “current” in your Ghost /path/to/ghost/content/themes directory, pointing at a working theme. (For example, from the themes directory: ln -s casper current.)
3. Navigate to the theme settings page of Ghost admin (http://localhost:2368/ghost/#/settings/design/change-theme), click installed, and activate the theme that says “(current)” after the title, if it isn’t already active.
4. Make sure the needed Unix utilities are installed: ln, find, basename, xargs and fzf. Likely, FZF is the only one you'll need to install (sudo apt install fzf). Note: while this symlink-updating approach should work on any OS, and many of the commands probably work pretty much the same way on MacOS (or maybe even Windows Subsystem for Linux 🤮), I have only tested them on Ubuntu Linux.

Now, we’re ready for fast theme switching. Change into the directory with your Ghost install (from here on, referred to as /path/to/ghost, which would be a terrible place for a Ghost install):

rm content/themes/current && ln -s $THEME content/themes/current

Here, $THEME is the directory (no path needed) of the theme you want to activate.

But this isn’t fancy enough yet. We want one command to do it all. Here it is:

ln -sfn "$(find content/themes -maxdepth 1 -type d -print | fzf | xargs basename)" content/themes/current

What this does is find directories that are direct children of the “themes” folder, pipe them to a fuzzy finder to let you pick, strip the path, and then update the existing symlink, passing the bare directory name as the target for ln.

Now, you can add something like this to your .bashrc, if you want it to be even more convenient:

alias gt='cd /path/to/ghost && ln -sfn "$(find content/themes -maxdepth 1 -type d -print | fzf | xargs basename)" content/themes/current'

Presto!

I found a used Pinebook Pro on Craigslist for $100 CAD, invented some flimsy excuse^[1], and bought it.

Then I went much farther down the rabbit hole than I expected, trying to get it functional.^[2]

Here are my first impressions of the journey to get (semi-) functional Linux and BSD systems installed on the Pinebook Pro.

What works out of the box

Here’s what Just Works™ with Manjaro KDE, the stock Pinebook Pro Linux distribution:

• It boots!
• Screen brightness keys
• Wifi
• Keyboard
• The battery charges
• USB
• Suspend (mostly)
• Privacy switches (though there’s no way to confirm)

What I like

• Hackable! Linux-first!! Open-hardware!!!
• Cheap ($100 CAD used)
• Excellent battery life
• Lovely magnesium alloy case
• Decent keyboard
• Nice screen
• It was my introduction to Arch linux / Manjaro, which was surprisingly pleasant:
  • various nice things are packaged for Arch linux that aren’t packaged for Debian/Ubuntu, including Sway, Helix, & Nyxt browser
  • Manjaro software installer GUI is very nice, and actually works (unlike Ubuntu’s new software store)
  • KDE Plasma, aside from how laggy it is on this little machine, is nice and polished, and Manjaro Sway is surprisingly slick, too.

The rough edges

• It ... is ... very ... slow ... to ... do ... … … anything. Even basic apps (like Manjaro system settings) hang frequently. Web browsing is barely usable. This is probably a combination of the underpowered specs (little ram, sub-SSD disk speed, lots of slow cores), and the fact that fewer things are optimized for Arm.

• The touchpad performance is extremely shoddy out of the box, especially at fine movements. This turned out to be partly a firmware issue (see below).

• Installing OpenBSD is a real headache (you’d need a UART serial console, rockchip DTB files, etc.) This is too bad because I kind of got the machine in order to fiddle with OpenBSD...

• Privacy keys are not hardware killswitches, they’re controlled by the keyboard firmware, which could be compromised.

• No dedicated delete key :/

• Having the power button as part of the keyboard is irritating; especially since it’s where the delete key should be on a full size keyboard, and I accidentally hit it all the time.

• It wasn’t possible to flash Tow Boot onto the SPI using the Tow Boot SD card installer until I had removed U-boot from the eMMC. (It might have been possible to dd Tow Boot onto the eMMC, but I figured the installer would be more reliable, so I got rid of U-Boot first.)

• My first attempt to overwrite eMMC with zeros using dd mysteriously failed. (Eventually either dd with /dev/urandom or else blkdiscard /dev/mmcblk2 seemed like it worked...)

• I was unable to connect to an external 4k monitor (Asus TUF) using Baseus USB-C hub, either on NetBSD or Manjaro

• And some other quirks that are probably Arch Linux / Manjaro specific:

  • Suspend failed one time. Thankfully, it provided lucid instructions on what command to run on a virtual terminal to get back in (Ctrl+Alt+f3, run loginctl unlock-session 2)
  • Setting language to en_GB somehow set keyboard layout to en_GB, even though I specified en_US keyboard layout. (This is probably a Manjaro thing, not Pinebook specific.)
  • Arm printer drivers didn’t work. It’s probably possible (by extracting and converting various pieces), but I gave up after wasting too much time trying to figure it out.
  • On first install of Manjaro, pacman (the Arch Linux package manager) had all kinds of GPG signing errors. I tried all the fixes I found mentioned on forums. Eventually, I knobbled the security features by editing pacman.conf, which fixed it; then I re-installed Manjaro, and that time it worked...

Here are some of the steps I used to get up and running:

How I tried to erase the SPI

First, install mtd-utils with sudo pacman -S mtd-utils then run sudo flash_erase /dev/mtd0 0 0 . (Not sure if this actually worked.)

Write Tow Boot installer to SD card

On my normal machine, I downloaded pine64-pinebookPro-2021.10-005.tar.xz from Tow Boot’s latest GitHub release (link), unzipped it, and, following the instructions, wrote the installer to an SD card with:

sudo dd if=spi.installer.img of=/dev/mmcblk0 bs=1M oflag=direct,sync status=progress

Useful things I installed right away:

pacman -S kupfer helix vim nyxt hdparm

Kupfer was unable to set its own hotkey (probably because Wayland), so I set up a keybinding through system settings.

Install sway & optional dependencies

pamac install sway bemenu dmenu foot i3status mako polkit swaybg swayidle swaylock waybar xorg-wayland

I think pamac is Manjaro’s more user-friendly wrapper around pacman?

Overclocking?!

I tried to use corectrl but it showed up blank. I tinkered with cpupower but even on the default governor the A72 cores were maxed out.

I have asked on Twitter, but as far as I can tell, unlike the Raspberry Pi, there’s no straightforward way to overclock this machine without re-compiling the kernel.

https://wiki.pine64.org/wiki/Overclocking

How to update your Manjaro system

sudo pacman -Syu

How I overwrote the eMMC (but not as securely as I’d like)

One or other or both of these seemed to work:

sudo dd if=/dev/urandom of=/dev/mmcblk2 bs=8192
sudo blkdiscard --secure /dev/mmcblk2

However, I wanted to use secure erase. Installed hdparm, but hdparm -I /dev/mmcblk2 just prints /dev/mmcblk2, so I assumed the eMMC didn’t support ATA secure erase.

However, the datasheet indicate that secure erase is supported.

pamac install mmc-utils

mmc erase secure-erase 0 9999999999999999 /dev/mmcblk2

... seemed to work. I also ran the command on
/dev/mmcblk2boot0 and /dev/mmcblk2boot1, which said it succeeded. I ran it on /dev/mmcblk2p1 and /dev/mmcblk2p4, after having run it on /dev/mmcblk2, and those failed (maybe because the partitions had already been secure-erased?)

I still don’t know if I have securely erased the eMMC, but I’ve tried enough things that it seems unlikely there are any but the most durable creepy-crawlies left on it. I can confirm that mmc erase did not work. BSD still tried to boot after it should have been gone...

How I flashed an installer to the eMMC

Successfully wrote BSD to eMMC (from Manjaro running off an SD card) with:

sudo cat '/run/media/pine/Data/install73-arm.img' | sudo dd of=/dev/mmcblk2 bs=1M status=progress conv=fsync

There was also talk of using manjaro-chroot -a or manjaro-arm-flasher, which I installed with pacman -S majaro-arm-flasher, but it wouldn’t run on Wayland, needed sudo, sudo -E didn’t work, and I'd forgotten about wsudo...

Run GUI apps with sudo in Wayland

This sometimes works, but is probably a bad idea:

sudo -E gnome-disks

See also: https://www.autodidacts.io/switching-to-sway-wayland-from-i3-x11-ubuntu/#makingitpossibletorungraphicalapplicationswithsudo

How to write Manjaro disk image to an SD card

sudo xzcat '/home/pine/Manjaro-ARM-kde-plasma-pbpro-23.02.img.xz' | sudo dd of=/dev/mmcblk1 bs=1M status=progress conv=fsync

How I tried and failed to fix the issues with Pacman

killall gpg-agent # (didn’t work)
sudo pkill gpg-agent
sudo rm -rf /etc/pacman.d/gnupg/
sudo pacman-key --init
sudo pacman-key --populate archlinux
sudo pacman -Sc
sudo pacman -Sy archlinux-keyring && sudo pacman -Su

How I updated the keyboard firmware, which significantly improved touchpad performance

I carefully followed the instructions in README to compile and use this keyboard firmware updater:

https://github.com/dragan-simic/pinebook-pro-keyboard-updater

It worked, and improved fine movements of touchpad.

NOTE: my usb keyboard did not work at the Tow Boot stage. Therefor, I had to open the case and switch off the eMMC. Thankfully, the USB keyboard worked to login and issue the remaining commands; otherwise I would have been hooped.

Attempts to connect to a 4K monitor over USB-C

I have an Asus VG289Q 4K monitor. It will talk to a Dell XPS 15 running Ubuntu via a Baseus JoyStar USB-C hub (though only at 30Hz). I have not been able to connect to the monitor using that hub from the Pinebook. On NetBSD, the hub doesn’t even light up. On Manjaro, the hub works for ethernet, but the HDMI signal does not reach the monitor. I have tried flipping the USB-C cable over, as some have suggested. I also ran across mention of running sudo pacman -Syyu linux-pinebookpro linux-pinebookpro-headers, however I get “target not found” errors for those packages.

I know that people have successfully run 4k monitors with the PineBook Pro, including, I believe, the model I have. A Baseus USB-C hub is listed as “good” in the hardware compatibility table. I’ve read that Manjaro used to distribute a patched kernel (5.7?), and then dropped the patch for the 5.8 kernel (?), which broke 4k monitor support (which relies on USB-C DisplayPort Alt Mode), and the fix never made it to mainline.

I’ve given up, for now.

Performance tuning

I followed some of the tips in this article; specifically, setting swappiness, vfs_cache_pressure, dirty_ratio, and dirty_background_ratio to the recommended values.

However, rather than in a single file as described (/etc/sysctl.conf), those setting are now set with individual configuration files in /etc/sysctl.d/. So I made a file for each parameter, putting vm.swappiness=100 in /etc/sysctl.d/swappiness.conf, and so forth.

Regarding OpenBSD on the Pinebook Pro

Installing OpenBSD on the Pinebook Pro can’t be done without using a UART serial console:

https://tomscii.sig7.se/2022/02/Guide-to-OpenBSD-on-the-PinebookPro

... or getting into some heavy and experimental patching:

https://tomscii.sig7.se/2022/03/Patches-for-OpenBSD-on-the-PinebookPro

Installing NetBSD

Well, if we can’t have nice things, maybe we can at least have semi-nice things? I decided to try to install some other BSD, and settled on NetBSD.

• Download from: https://nycdn.netbsd.org/pub/arm/

• Replace xzcat with gunzip -c, and the same dd command used to write the Manjaro installer to the eMMC works:

gunzip -c /path/to/installer.img.gz | sudo dd of=/dev/mmcblk1 bs=1M status=progress conv=fsync

I went away, and when I came back, it was installed, and at login. Hit enter, type startx, and we’re golden!

How to use it was another question...

Using NetBSD

Enable ethernet:

ifconfig

... will print out various things. In my case, there was “lo” and “cdce0”:

ifconfig cdce0 up

... Internet connection established! (I tested it with dig autodidacts.io; you might want to use a site with higher uptime...)

Wifi drivers are not available for the Pinebook Pro.

Also, the USB 3 port on the left side does not seem to work on NetBSD.

Installing packages:

vi /root/.profile

As I recall, I uncommented the “PKG_SRC...” line, but that didn't do anything, so then I ran export PKG_SRC= with the value from the line in /root/.profile. (only works with http?)

How to upgrade NetBSD system (can be quite slow, if it’s been a while):

pkgin update
pkgin upgrade

Add some useful apps:

pkg_add vim git firefox helix zoxide gnome-terminal bash nushell fzf pkgin nautilus cryptsetup pandoc 

Unsolved:

• There was a warning about Firefox security, and something about a timestamp, which I never got to the bottom of. Firefox was too heavy anyway.
• Copy and paste works in gnome terminal, but not in uxterm.
• I was able to compile whisper.cpp text-to-speech on NetBSD and batch-transcribe audio overnight using the medium model. I was not able to get SDL2 support working for real-time streaming transcription (even though I think I installed the right SDL2 packages).
• I tried to install the following apps, and found them not available: which, locate, kupfer, gnome-do, albert, lshw, lsblk, nyxt, ulauncher.

Further reading: https://wiki.netbsd.org/laptops/

XFCE on NetBSD

Install it with:

pkgin install xfce4 xfce4-extras 

(I also installed the cpugraph and system load plugins, and some others.)

Start it with:

startxfce4

It seems to basically work!

How to mount a USB drive

# Figure out which drive to mount. In this case, I’m searching for info about an ADATA thumbdrive:
dmesg | grep ADATA

# The above command revealed that it was /dev/sd0. “a” is the partition? I just guessed that part and it worked.
mount_msdos /dev/sd0a /mnt

# When you’re done, unmount drive with:
umount /dev/sd0a

But what about if you're using an encrypted drive? In that case, you’ll need to pkg_add cryptsetup (included in the above list of programs to install), then:

cryptsetup luksOpen /dev/sdb1 luks_device
mount /dev/mapper/luks_device /mnt

I never got this to work AFAIK.

How to set backlight brightness:

sysctl hw.pwmbacklight0
# hw.pwmbacklight0.levels = 0 30 40 50 60 70 80 90 100
hw.pwmbacklight0.level = 80

Default shell?

The default shell, strangely, is sh, rather than bash. sh doesn’t seem to have Ctrl+R reverse history search, which I depend upon for rapid use of the command line.

So, one of the first things I did was to install bash, and, just for kicks, nushell (which I’d tried before). I run nu every time I open a terminal, since I haven’t bothered to set the default shell yet.

First impression of NetBSD

Let me just say, I liked NetBSD as soon as it booted (after all, it had booted), even though I didn’t know how to get it to do anything. When I’d briefly experimented with OpenBSD on a ThinkPad, the install process was confusing, and nothing much worked.

To be very untechnical, NetBSD felt small and light and stable. If Manjaro was a very smart, powerful person who was overcaffeinated and under-rested and having a very bad hair day … NetBSD was an aging but good-natured accountant, quietly and placidly adding sums by hand over its spectacles, while sipping tea. NetBSD isn’t fast — but it doesn’t hang when you ask it to do something that taxes the hardware specs (like, in this case, opening Firefox). It just does it very slowly. And I like that a lot.

That said, it’s not for those who want an out-of-the box experience and nice GUIs. Wifi does not work, and I haven’t found a way to get it to. Screen brightness keys don’t work. Suspend does not work (it turns off the screen, that’s all, when the lid is closed).

Conclusion

The Pinebook Pro offers appealing hardware at a very low price. However, the combination of Arm architecture, underpowered specs, and poor software support for the hardware make it frustrating to get things working, or be productive in any way.

But it’s usable for writing text, and a fun way to learn about different Linux & BSD operating systems.

I propose a new standard for the blogosphere, along the lines of Derek Sivers’s “Now” page: the Things I don’t know how to do list.

Criteria:

1. Maintain a more or less up-to-date list of things that you don’t know how to do, and would like to learn.
   • These should be things that you have tried to learn how to do, and failed: things that are small enough to be achievable, but not the kind of thing that can be solved easily with a search engine.
   • Optionally, include information about how hard you’ve worked on the problem already, and what solutions you tried that didn’t work.
2. Ideally, put a link in the footer of your site to the page. I propose the interrobang (‽), since it’s not widely used, and conveys the curiosity and frustration of unsolved problems that should be solvable but have so far been uncooperative.
3. When one of the problems is solved, strike out the question and add the solution, or a link to the solution. (It’s occasionally permissible to remove answered questions that make the blogger look really dumb.)
4. Optionally, include a list of expertise that you could share with anyone who helps you solve one of the problems
5. I propose the page title “Things I don’t know how to do”, and the URL /intractable.

Why bother?

The people reading blogs know all kinds of things the blogger doesn’t. Blogs have high surface area. By maintaining a list, the blogger opens up the possibility of being helped.

But it also gives a low friction way for readers to get in touch with bloggers. It creates a passive way of helping each other solve obscure but tractable problems, and meet other friendly internet citizens.

I’ll start:

• Fall asleep in less than ten minutes, when not sleep-deprived. (I have tried 4-7-8 breathing, and the 2-minute military relaxation techniques, as well as following as much as possible of the usual advice on “sleep hygiene”.)

• Connect a USB keyboard to a Kobo e-reader, or use it as an external monitor (connected over USB, rather than through the internet), without supermassive headache (I have done a lot of Googling, and started an Ask HN thread.)

• See what LibreOffice Calc formula or function is slowing down the worksheet. (I have Googled this, but not done thorough research.)

• Get Nouveau + Wayland to work after upgrading to Ubuntu 22.04 LTS, on a Thinkpad T430 with GF108M [NVS 5400M] graphics. (Now, only the Nvdia 390 drivers + Xorg work. I have tried the usual sudo apt purge *nvidia* without success.)

• Reliably prevent nosebleeds (I’ve tried: Vitamin C, Calendula.)

• Find any rigorous research on optimizing handwriting speed, legibility, and comfort. (I have Googled, and started an Ask HN thread.)

• Figure out how to setup Discourse when running the Bitnami Discourse Docker container on Fly. (The step I am missing is the “executing a shell inside the container”, I believe. When I SSH into fly and run the bundle rake command from /opt/bitnami/discourse, it fails. More info here.)

• Find a free video tutorial series teaching intermediate-level Waltz steps.

• View waveform/stem of audio files in VLC or MPV (or better yet, in Nautilus!), the way that phone audio recorders show it (allowing you to easily skip to the salient section, and get a visual overview of the track.)

• Install OpenBSD on the Pinebook Pro without taking it apart and having a UART serial connector (I don’t believe it’s possible, but maybe it will be soon.)

• Find a website for verbose, idiot-proof, almost page-by-page user-contributed summaries of (fiction) books and short stories. (I often find myself frustrated that I have obviously missed the point of some obscure science fiction story, and Google it, finding reviews, but no blow-by-blow plot explainer. An obvious example: I remain embarrassingly clueless about the symbolism in Voyage to Arcturus.)

• Discover a chemical that reverses the bleaching effect Hydrogen Peroxide has on Diamine Registrars Blue-Black waterproof iron gall fountain pen ink.

• Find any Danish literary magazines that pay writers and accept fiction submissions in English.

My skills

» Click to expand

I might be able to help with problems related to the topics covered on this blog: Linux, the command line, writing fiction & non-fiction, music & music theory, distance running, philosophy, DIY neuroscience, chemistry, spreadsheets, memory enhancement, health & nutrition, electronics troubleshooting, web design, et cetera.

A few things I know about, but haven’t written about yet, include how to:

• Cure an intractable seasonal rash with diet
• Use rsync to backup ereaders, or keep an MP3 player synched with music on a laptop
• Run NetBSD on the Pinebook Pro
• Use Regex to correct common Tesseract OCR errors
• Publish short fiction in magazines
• Make a highly-satisfying submissions tracking & cover-letter-generating spreadsheet
• Use the Nashville Number System
• Bulk transcribe audio memos with Whisper.cpp
• Get people to read my lousy blog posts
• Negotiate
• Live life without a smartphone

And here are some notes I have written for grug myself which I can share:

Join the club!

If you start a “Things I don’t know how to do” list on your blog, please email me the link or drop it in the comments below so I can read it — and look for anything I can help with.

This is yet another small companion piece to my Ghost on Fly tutorials.

In my last tutorial in the series, I showed how to back up your Ghost blog hosted on Fly.io over SFTP.

SFTP does the job, but transferring all those images and files every time is a waste of bandwidth — and pig-slow.

Rsync is an amazing command line file synchronization utility, and a much better tool for the job.

It took me a while to figure out how to scp file to and from Fly VM persistent volumes. Once I’d figured that out^[1], swapping scp for rsync was simple.

Here’s how to back up your blog in seconds, rather than minutes, without sending eight years worth of uploads over the wire each time.

In one terminal, run:

flyctl proxy 10022:22

In another, SSH in:

fly ssh console

Install rsync on the remote (which is running Alpine Linux, if you followed my tutorial):

apk add rsync

Back on your local machine, run:

ssh-agent bash
ssh-keygen -f "/home/USERNAME/.ssh/known_hosts" -R "[localhost]:10022"
flyctl ssh issue --agent

After that, run the backup! To rsync Ghost’s content files from the persistent volume:

rsync -aviz -e 'ssh -p 10022' root@localhost:/var/lib/ghost/content/ .

I’m using the -a (--archive) flag with -z (--compress), with -v (--verbose) -i (--itemize-changes).

It probably doesn’t hurt to run it with the --dry-run flag first, to make sure everything’s working as expected. 😇

In the unlikely event that you found this tutorial so helpful and thrilling that you’d like to support me, you can sign up for Ghost(Pro) with my referral link, or buy one of my premium themes Or, if you’re like me, host your blog for free using one of my free themes instead, while thinking grateful thoughts.

Sometimes, you just want to link to a specific segment of a YouTube video. Deep-linking to a specific start time in a YouTube video is fairly common — but sometimes specifying an end-time is as (or more!) important.

After much fruitless Googling, I’ve found a way to do this:

1. Use the embed URL (https://youtube.com/embed/ID) rather than the watch URL (https://youtube.com/watch?v=ID or https://youtube.com/watch/ID)

2. Specify the start time, end time, or both, using the URL parameters start and end. The units are seconds. I haven't figured out a way to use other units but it might be possible.

Here are seven seconds from the middle of a Joel Spolsky talk:

https://www.youtube.com/embed/0nbkaYsR94c?start=41&end=48

(When you use embed URLs as links, it’s just the video — no sidebars or comments. In my view, that's a good thing.)

Here’s an actual embed of the above embed URL, so you can see the query parameters at work without leaving the page:

Here’s the same video, starting one minute in:

https://www.youtube.com/embed/0nbkaYsR94c?start=60

Here’s just the first minute of the talk:

https://www.youtube.com/embed/0nbkaYsR94c?end=60

[For those who aren’t familiar with URL query strings: a question mark goes after the URL and before the first query parameter, and additional key-value pairs are separated with an ampersand.]

I’d run across mention of YouTube allowing one to link to a segment of a video using ?start=30&end=60 (or in some cases stop rather than end) in various articles, but these parameters don’t work on the watch URL. When sharing a regular YouTube link, start=30 turns into t=30s and the stop or end parameter gets thrown out.

At first, I assumed YouTube had removed this useful functionality. But the docs for the embedded player still mentioned the end parameter, so I tried using them on the embed URL when not embedded — and it worked!

I hope this helps anyone who has the same question I had.