I've been using Linux as my main/only operating system for my entire life. While I would like to brag that Linux is more stable than Mac, the simple fact is that it isn’t. This is sad, because it’s great in every other way, and the bugs are mostly fairly surface-level, yet give it a patchy reputation among normal users.

Because I spend a lot of time in the terminal, and all my friends also run Linux, I often get called in to help when their systems have issues. Often, I fail to solve the problem, which is frustrating. So over the past few years I have taken some time to learn more about recovering from all the issues that need recovering from, and collecting my techniques. Here are some of the most useful tools and approaches I have found.

Over the years, I have used Knoppix, Ubuntu, Linux Mint, Manjaro, NetBSD, OpenBSD, and others. These tips are mostly for Ubuntu, Linux Mint, and other Debian-based distros. I have used Xfce, Compiz, CWM, Cinnamon, Sway, Mate, and Gnome. These are mostly about Gnome and Cinnamon.

Note: this article is a living document that I intend to add to as I learn more and have more time. If you have a specific problem, write in or drop a comment, and I will prioritize that section.

This post pairs well with the more high-level advice in Troubleshooting: A Skill That Never Goes Obsolete.

If a program has crashed

killall PROGRAM_NAME (usually, killall firefox)

Scary black screen

If all you see is text, there's a goodish chance that it’s a problem with the graphics drivers. Try re-installing them. If you're using the proprietary Nvidia drivers (which are a nightmare in my experience) try using the open Nouveau driver. Also try switching to a different Nvidia driver if one is available. Also, try fully purging and re-installing Nvidia drivers. If you're using the Nouveau driver, try switching to Nvidia drivers. If you're using Xorg, try switching to Wayland. If you're using Wayland, try switching to Xorg.

sudo apt purge *nvidia* # potentially dangerous
sudo ubuntu-drivers install # follow prompts

If you’re using a Macbook, good luck. In my experience, despite what everyone says, Macbooks have some of the worst Linux support. The four Macbooks I’ve encountered that were running Linux were all unmitigated problem cases, and graphics drivers were the weakest point. Once I spent seven hours getting a friend’s Macbook back to graphical functionality after an Ubuntu upgrade. I breathe a sigh of relief when Macbooks finally conk for good, and ardently recommend the owner upgrade to a ThinkPad if they want me to keep doing tech support.

Google the issue + the model of your laptop. Often you can edit /etc/default/grub and add something to the GRUB_CMDLINE_LINUX_DEFAULT=... line. Often that something is nomodeset. This is also often how I fix issues with screen brightness and backlight.

Once you get back to functionality, you can try to figure out why the specific configuration you had before broke, by looking in the logs. Or if you're happy with the new config, just forget about it and use what works.

If the mouse slows down and gets jittery, and the fans rev up, and the I/O light blinks fast or goes solid, and then the system fully freezes

This usually means that:

1. Somebody opened too many tabs in Firefox (likely something JavaScript heavy like Reddit, YouTube, or Facebook)
2. The computer ran out of RAM
3. It started swapping
4. Now it’s crashed

We can stop this at any point in the chain, but the farther up, the better.

Don’t open so many tabs. Or, use about:unloads to unload the heaviest tabs when you start to notice slow downs. If you catch it when the mouse starts to lag, immediately hit alt-f4 to quit the most resource intensive app that won't lose work (usually your browser), then wait. Do not press close over and over again: it will just make the computer mad. If you manage to get a heavy app closed before the system becomes unresponsive, it will likely recover, and you can take further steps.

You can also start the culprit app (usually a browser or media transcoding workload or LLM) with a systemd memory limit, which generally prevents it from causing this issue. (Or you can try to renice the app, but this doesn't seem to be as effective.)

Here’s how to run an app with a 4GB memory limit:

systemd-run --scope -p MemoryMax=4G --user COMMAND_NAME

(Change COMMAND_NAME to firefox, ffmpeg, llama-server, or whatever is taking the system down with it.)

There’s also ulimit, but I haven't used it as much.

The simplest fix: BUY MORE RAM. Years ago, going from 8gb to 12gb of RAM on my ThinkPad went from daily seizures of this type to annual seizures. Basically cured!

Bad ram (or badly-seated ram) can also be the culprit.

On servers with limited RAM, adding swap can save the day, but I find that when a desktop Linux system starts swapping, that’s pretty much the end of the good times and the beginning of the bad times.

An SSD might help, because swap speed will be closer to RAM speed. (An SSD is one of the most effective ways to make your computer go faster, other than adding more RAM.)

There are various settings related to swapping behaviour ("swappiness") and overcommiting memory that may help (see here), but I haven't actually used them on desktop Linux.

If you don't manage to get the app closed, try to get to a console. Try: opening a terminal (likely won't work), alt+f2 (also likely won't work, because it's part of the window manager which is crashed), and then dropping to a virtual console with Ctrl+Alt+f2. Press each key slowly, on after the other, holding down the previous keys: the response time is way slower than a normal keyboard. The exact f-key varies: generally Ctrl+Alt+f1 gets you to a login session screen, which you don't want, and the main session is on Ctrl+Alt+f5 or Ctrl+Alt+f7, so I find that f2 or f3 are most likely to get me to a virtual terminal.

Enter username and password. Now you should be able to run killall firefox (or ffmpeg, or ollama, or whisper-stream, or whatever it is). You can also inspect what's crashed with top or htop. htop is good stuff! (and also makes it easy to send stop signals with f9 and then a number; in this case, likely 9) If you can get to the virtual console, you can usually can rescue an out-of-ram freeze caused by a specific app.

But the most powerful technique I've found for this is enabling Magic System Request keys, and then using Ctrl+Alt+PrintScr+f at the first sign of trouble to activate the OOM_KILLER to terminate the most resource-intensive process (usually a firefox tab). It works wonders.

Check your current sysrq value with:

cat /proc/sys/kernel/sysrq

Edit the kernel.sysrq numerical value in the alphabetically last file listed by this command:

ls /etc/sysctl.d/ | xargs -I{} grep -l kernel.sysrq /etc/sysctl.d/"{}"

(After rebooting, check that the value stuck using the previous command.)

Then you can see what it quit with:

dmesg -T | egrep -i 'killed process'

... and hope it wasn’t anything important.

If it’s the desktop environment that has crashed

If you’re running Cinnamon (Linux Mint’s default desktop environment), try running this from a virtual console:

DISPLAY=:0 cinnamon --replace

If you’re using Gnome (on Xorg), you can try these, but I have never managed to restart a Gnome session without it quitting all open apps, and losing data:

pkill -3 gnome-shell
sudo pkill -9 ^gnome-shell

If you know how to, let me know!

If you’re unclear about the difference between a window manager, a desktop environment, a display server, and a distro(bution) are

I’m a bit fuzzy on this, too! Specifically, the where the line is between a window manager and a desktop environment. But the general stack goes, from low-level to high level:

Operating System Family (ie, Linux, BSD, Mac, Windows) → Distro (ie, Linux Mint, Ubuntu, NetBSD) → Display Server (X Window System, Wayland) → Desktop Environment (ie, Xfce, Gnome, Cinnamon, KDE) → Window Manager (Compiz, Openbox, Sway, i3) → Theme & Settings (these days I just use the default, but I used to regularly break my system trying to make it look fancy).

If your screen goes all colorful or develops lines and weird patterns

If you’re using an external monitor, likely the cable connector isn’t seated correctly, or the contacts are dirty.

If it’s a laptop, it’s most likely an issue with the graphics drivers, or the screen connector. I have had this happen a few times, usually related to GPU intensive workloads and suspend, but I haven’t found the root cause, because I encounter it so rarely. All I know is that restarting Cinnamon from a virtual console doesn’t help.

If it has fully crashed and rebooted on its own

Oof, this is serious. In my experience this is fairly rare. I/O issues on peripherals have caused this in the past (ie, excessive power draw from a dying hard drive). Check the logs for insight.

Suspend issues

These often have to do with the graphics drivers, and the specific hardware. Sometimes adding grub command line options helps. Often changing drivers helps.

If you want to know what a program is doing

sudo strace -p$(pidof firefox) -e trace=write -s1000 -fp 18386 2>&1 | grep -o '".\+[^"]"'

(replace firefox with the name of the program you want to know about)

If a program has a bug, this is useful for viewing the error message / stack trace, without quitting the program and re-running it from terminal.

If you want to know what is hogging all the resources

My go-to is the wonderful htop. top is installed by default. glances and btop can also be helpful.

For figuring out suspiciously heavy network or disk write activity, try iotop and nethogs. Also, powertop, if you want to tune battery life.

If you want to know whether a script actually quit or crashed and left pieces churning along in the background

Run htop and search with f3. I often find that a script called some process, then crashed, and left a heavy process still running. This mostly happens when you're developing (ie, for me, Bash/Python scripts that called Whisper C++ or Selenium), or using really bad quality software. Also, viruses.

If you're stuck in Vim

:q!

If the screen is black

Is the backlight on?

Try plugging in an external monitor. This will tell you whether it's your graphics drivers, or hardware.

If external monitor works, it could be your backlight (common), or your screen connector. (These are both pretty easy to fix if you have a repairable laptop like a thinkpad.) Or, it could be the LCD panel itself.

It also could be a hardware/driver issue that can be fixed with the grub command line options.

If the Wifi doesn't work

Try wired internet. If it doesn’t work either, the problem isn’t the Wifi drivers.

Make sure you can get internet from the network on a different device.

Check what the output of ifconfig looks like.

Try running sudo service NetworkManager restart (fixes it surprisingly often!)

If your WiFi hotspot has no internet in it

Maybe it’s your VPN, or firewall. But in my case, it was broadcasting on channel 13, which is only supported in Japan.

I was able to fix it with:

nmcli con modify "Hotspot" 802-11-wireless.band bg # band must be set. Options, in my case, were "a" and "bg". I do not know the difference, but "bg" worked.
nmcli con modify "Hotspot" 802-11-wireless.channel 6 # set the channel
nmcli con up "Hotspot" # bring it up

(Change "Hotspot" to your network name or UUID.)

If the printer won’t print

Try rebooting the printer.

Try a different USB port!

Try clearing the print queue and re-sending the job (very often works!)

Make sure you sent it to the right printer (there are often multiple entries for the same printer, only one of which works.)

Try running sudo service cups restart. This works quite often.

If your mouse or touchpad has stopped moving

Open terminal and run this command to reload the drivers:

sudo modprobe -r psmouse && sudo modprobe psmouse

This usually fixes it.

If it doesn’t: try switching to Synaptics drivers if you’re using libinput drivers, and to libinput drivers if you’re using Synaptics drivers.

If you're having trouble installing software

Try running sudo apt update to make sure you have up-to-date package cache. Look at the error message when you run sudo apt upgrade or sudo apt install PACKAGE_NAME_HERE. The error messages here are often more helpful than the ones you get from the GUI.

Remove old PPAs etc that you don't need any more.

Try running sudo apt install --fix-missing; this usually fixes it.

If an app installed with snap has mysteriously disappeared, it might be in the process of being updated.

If you get Error Splicing File errors

Your external media or internal HDD are likely full! This is a terrifying situation that happens all the time. Many things will cease to function, because they require tmp space, even things that you would expect to keep working.

Find out if your internal root disk is full with:

df -h /

Here are some commands for finding big files that you might be able to delete:

du -a 2>/dev/null | sort -n -r

du --max-depth=1 /path | sort -r -k1,1n

find /home -type f -exec du -s {} \; | sort -r -k1,1n | head

du -a -h /path | sort -h -r | head -n 10

At the time of writing, my 1TB internal drive is 100% full, with 7gb free. Wish me luck!

If the issue is with external media (thumbdrive, HDD)

If it doesn't show up in file manager, check in Gnome Disks; if it doesn't show up there, check in the logs (and also try it on a different computer, and a different USB port, and try a known-good flashdrive/HDD on the computer in the same USB port to isolate the problem).

Errors tend to show up in dmesg. Grep for the manufacturer name (ie ADATA), or for usb or device. Sometimes the actual device is rebranded, so it will say one thing, but will show up as a bigger manufacturer (often Toshiba).

This will help get the /dev/sdX, and then you can go from there.

If a SD card / flashdrive got corrupted

If it is critical, take it to a specialist. This will cost a bundle. Otherwise:

• Create a disk image with ddrescue
• Do all further operations on a copy of the disk image, not on the disk image, and definitely not on the device itself (read/write intensive operations might damage it more)
• Use fsck to repair corrupted file systems. Often it works. (It sometimes leaves garbage fragments of deleted files.)
• Use photorec to recover photos and videos that fsck can't recover
• Use untrunc to piece back together corrupted videos (as long as you have a good video from the same camera)
• ^^^ these two tools can save rashers and rashers of bacon. Consider donating to the maintainers (and also, the bloggers who told you about them)

Free advice: never buy no-name-brand ultra-good-price storage from Ebay or Amazon. Never. When it comes to storage, cheap can be very, very expensive.

Check the logs

The logs are the troubleshooter’s best friend. On Linux, the good stuff is mostly in /var/log

There, you will likely find logs for all kinds of things. The most interesting ones:

dmesg (includes peripherals, startup stuff. #1 stop)

syslog (#2 stop)

kern.log (kernel stuff, obviously)

Xorg.log (X Window System)

Probably there will be multiple files for each log, because logs are rotated. So you'll have dmesg, dmesg.0, dmesg.1.gz, dmesg.2.gz, etc, etc.

Five ways to view a log:

1. cat filename (print whole thing to console. most basic)
2. open in a text editor vim filename. Useful for searching, reading at leisure, filtering, etc. Some text editors will choke on very large files (gedit, for example). Vim is pretty good at opening gigantic text files.
3. view top of bottom part with tail or head (ie, tail -n 10 filename to view last ten lines), or watch a file live with tail -f filename, then do the thing that might cause the error message, and see what gets output.
4. search in a file grep -i fail syslog or in the whole log directory (which tends to be slow) grep -i fail
5. journalctl is the fancy new thing that I don't really know how to use. It's supposedly great. Try it with journalctl -u cron or journalctl --since=-1hour. It can filter by process ID, executable name, date (including since-last-boot), urgency, and more
6. Bonus option: print it out on a thousand sheets of paper like a real hacker and then inspect it with a magnifying glass

The main things I do:

1. Just browse through all the logs and look for something that looks suspicious, or related to the problem
2. Tail the most-likely-to-be-relevant log with -f (follow) and then do the thing that triggers the bug and see what comes out of the log (very effective if the bug is reproducible, ie, an external drive won't mount, or a program crashes on startup)
3. Explicitly grep for error messages. These can take several forms. Sometimes it says Error, and sometimes it says Err: ... sometimes it says Failure to X and sometimes it says Failed to start Y. Other watchwords are "segfault", "panic", "fatal", "crash", (core) "dumped". You can find them all with:

grep -i -E "fail|fatal|err|panic|segfault|segmentation fault|dumped|core dumped|killed|warn|bad" syslog # grep case insensitive with extended regex for any of the following

Befriend the terminal

You won't get far troubleshooting Linux crashes without getting used to running commands.

Command history (up arrow) and completion (tab) and reverse history search (ctrl+r) and copying-and-pasting (ctrl+shift+c and ctrl+shift+v instead of ctrl+c and ctrl+v) are essential.

Also, normal keystrokes like Ctrl+Z and Ctrl+S do weird things in the terminal. Avoid.

Learn a console text editor

You will also want to get a decent grasp of a fully console-based text editor that is installed everywhere – Vi(m) or Nano. Nano is simpler, Vim is more powerful.

Just practice editing files with them for fun (write me an email in one!), and then you'll know how to open files, navigate around, edit, save, and quit, when it matters.

Also:

System files generally need to be opened with sudo (ie, sudo vim path/to/file.conf or sudoedit path/to/file.conf). If you don't, you might make all your changes and then not be able to save!

If you're in Vim when this happens, you can use :w !sudo tee %. (In Helix, it’s %:pipe-to sudo tee <Ctrl-r>%, but heed this warning about the importance of that first percentage sign to select all.)

General advice

Search the forums.

If necessary, post to the forums. Try to do your homework first. If people trying to help you are rude in their replies, try to ignore the tone and make use of the advice they give, be polite and grateful, and maybe do more homework next time.

The Arch Linux wiki is helpful even you don’t run Arch. Arch users have encountered nearly every bug under the sun. It tends to be pretty technical, though.

Reboot the machine.

Always backup first.

Try to understand commands before running them (but in practice sometimes you have to fly blind).

Unplug anything you don't need plugged in.

Try to figure out what changed that might have triggered the problem.

Beware "Partial Upgrades".

Do a fresh install from time to time.

Write down the exact time the problem occurred (so you can match it to log timestamps). Also, make sure to account for potential timezone differences: sometimes logs are in UTC.

Read my essay on troubleshooting.

Read the docs/manual for the software in question.

Keep trying.

Conclusion

If this article helped you, share it with a friend. If it didn’t help you, share it with an enemy. <3

If you know more than me, send me corrections and/or invective. If you know less than me, send me money and/or praise.

If you have an idea of how to fix the problem, try it. If you have no idea how to fix the problem, look it up. If the idea you had generated a new error message, you are either getting closer or farther away from solving the problem. Google the new error message.

If nothing works, keep trying. If everything works, switch to BSD.

EOF