Vol. 1 · No. 002   Home network edition

Today I Learned

All the bugs that are fit to print

Case file No. 002 · Home network · AirPrint

The Case of the Printer Nobody Could Reach

My iPhone could see my home printer, but it hadn’t printed a page in 127 days. Here’s how I tracked down why, what finally fixed it, and how to check your own setup when AirPrint goes quiet.

TL;DR

My iPhone could see my printer but couldn’t print to it. Five problems were stacked up: a changed printer address had paused the CUPS queue, the container’s print backend crashed, the printer’s raw port needed its own driver, and two Bonjour servers on my Synology NAS, one of them in a host-networked Docker container, kept renaming each other because they published different addresses for the same name. The fix that ended it: let the NAS own its name, and stop the container’s Avahi publishing one (publish-addresses=no).

Related to

  • Bonjour / mDNS
  • AirPrint
  • CUPS
  • Docker networking
  • Synology NAS
  • Home lab

What you’ll learn

  1. How your phone finds a printer by name when there’s no DNS server (mDNS and DNS-SD).
  2. Why two Bonjour servers on one machine keep renaming each other, and the three lines that stop it.
  3. What makes a printer “AirPrint”, and how a NAS can add it to a printer that doesn’t have it.
  4. A step-by-step checklist for a printer that shows up but won’t print.

Who it’s for

Anyone sharing a printer from a NAS or a Raspberry Pi, running Avahi in Docker, or wondering why AirPrint went quiet.

Level: intermediate. You’ll want to be comfortable in a terminal.

The sequel · No. 001 shows how my NAS gives this printer AirPrint in the first place

At 21:03 on a Monday night, my Epson L3250 printed a photo from my iPhone. An ordinary photo. It was the first page my phone had printed since the middle of June.

For 127 days, nothing had complained. The printer was on. My NAS, which relays my iPhone’s jobs to the printer because the Epson can’t do AirPrint by itself, was running. My phone could even see the printer in its print menu. Every job went in, and nothing came out.

I worked it out in one evening. It turned out to be five problems stacked on top of each other, and the one at the bottom wasn’t about printing at all. Here’s the trail, dead ends included, and what you can take from each step.

So where were my jobs going?

How I worked through it

Cold for 127 days, cracked in one evening

  1. incidentMy printer’s address changed from .2 to .7 when the router handed out a new one. CUPS kept sending to .2, failed once, and paused the queue for good, because it was set to stop the printer on any error. Nothing told me.LessonGive printers a fixed address with a DHCP reservation, and set CUPS to retry instead of stopping.

  2. twistContainer Manager quietly auto-updated my AirPrint container to version 2.x.LessonPin the image version of anything you depend on at home.

  3. progressI pointed the queue at the new address and told it to retry instead of giving up.

  4. incidentNow every job failed with “Backend failed”. A saved log named the culprit: CUPS’s ipp backend crashed with status -139, signal 11, inside the Alpine container.Lesson“Backend failed” means one of CUPS’s own helpers died. Read error_log before you blame the printer.

  5. dead endI tried going around the crashing backend by sending raw data straight to port 9100. The printer printed pages of gibberish, and kept going.LessonPort 9100 passes bytes straight through, so the data has to be in the printer’s own language already.

  6. progressI added Epson’s own ESC/P-R driver to the container. Now the NAS spoke the printer’s native language.

  7. incidentMy phone could see the printer but not connect. The advert pointed at cups-airprint-3.local, a name with no address: the 2.x image runs its own Bonjour server under its own name.LessonIf a printer shows up but won’t connect, check that the host in its advert actually resolves.

  8. dead endI pinned the image back to 1.2.0, the March version. The container’s Bonjour server started renaming itself anyway: nas-16, -50, -58.

  9. the clueI found my NAS’s own Bonjour server running as nas-319.local. Something had made it rename itself 319 times.LessonA name ending in a number like -2 or -319 means two devices are fighting over it.

  10. progressI restarted the NAS’s Avahi. My iPhone printed a photo.

  11. closedI stopped the container publishing a name or addresses. Both servers settled on nas.local, with no number.

The break

A name doesn’t reach -319 by accident

At 20:59 I stopped looking at the printer. My NAS’s process list showed its own Bonjour server, Avahi, running under the name nas-319.local. Its real name is nas.local, and nas.local no longer answered at all.

Bonjour adds -2, -3, -4 to a name when it finds someone else already using it. To reach -319, my NAS had lost an argument about its own name 319 times in the two weeks since its last reboot.

So I had a new question: who else on my network was calling themselves nas?

Wanted

for answering to a name that wasn’t theirs

“nas”

Names left behind
nas-2, -3, -4 … all the way to -319
Last seen
UDP port 5353, address 224.0.0.251
Method
Claims a name already in use, then politely renames. Again and again.

Reward: one printed photo

Spoiler: there was no thief. See Exhibit C.

Exhibit A · what you need to know first

Looking up a name with no DNS server

To see what went wrong, you need one idea about home networks. Bonjour is Apple’s name for two open standards working together: mDNS for looking up names and DNS-SD for finding services. Avahi is the Linux implementation of the same two standards. My NAS runs Avahi, my AirPrint container ran a second copy, and your iPhone runs Bonjour. They all speak the same packets.

On the internet, your phone asks a DNS server for an address. At home there is no server that knows your devices, so mDNS asks everyone at once. The question goes to 224.0.0.251 on UDP port 5353, which every device on the network listens to. Whoever owns the name answers. Names ending in .local are the signal to do it this way.

PTR · what exists SRV · where to connect TXT · what it can do A · which IP
Exhibit A · How a name is found
home LAN · 10.0.0.0/24 iPhone 10.0.0.31 ① “Who is nas.local?” sent to 224.0.0.251 : 5353, so every device hears it Mac .27 · stays quiet Router .2 · stays quiet Epson L3250 .7 · stays quiet nas owns the name ② “That’s me: 10.0.0.5 and 10.0.0.6”
One question, one answer. The question reaches every device on the segment; only the device that owns nas.local replies. Your NAS answers with two addresses because it has two network ports.
Exhibit B · how a phone picks a printer

Finding a printer, not just a name

DNS-SD reuses ordinary DNS record types to describe services. Your iPhone follows a chain of four lookups, each answering the next question. These are the real records on my network.

Exhibit B · The record chain
1 · which printers? 2 · where do I connect? 3 · what can it do? 4 · what is the IP? PTR _ipp._tcp.local points to AirPrint EPSON_L3250 @ nas the name in the print menu SRV host nas.local port 631 631 is the IPP port TXT rp=printers/EPSON_L3250 URF=none pdl=…pdf,…image/urf… Color=T key=value capability list A nas.local resolves to 10.0.0.5 10.0.0.6 published by DSM’s Avahi then the iPhone opens http://10.0.0.5:631/printers/EPSON_L3250 and sends the job over IPP
Each record answers one question. The SRV record names a host instead of an IP, which is why the A record matters so much: if nobody publishes an address for nas.local, the chain breaks at step 4 even though the printer still shows up in the list from step 1.

Why my Epson needed an accomplice

Showing up in the list isn’t enough. iOS also checks for the _universal subtype, a URF key in the TXT record, and a document format it can send. On its own, my Epson fails two of the four checks, so the iPhone leaves it out of the print menu. The NAS relay passes all four. If your printer isn’t AirPrint, this is how you give it AirPrint.

Check the iPhone makesEpson’s own advertNAS advert
Listed under _ipp._tcpyesyes
_universal subtypeyesyes
URF key in TXTmissingURF=none
Accepts a format iOS sendspwg-raster, escprpdf, image/urf
ResultNot shown as AirPrintAirPrint printer
Exhibit C · the reveal

There was no thief

With no server in charge, devices claim names themselves. Before using a name, a device probes: “does anyone own this already?” If someone answers with exactly the same records, that’s fine. If the answer is different, the device assumes a clash, adds -2 to its name, and tries again.

My AirPrint container shared the NAS’s own network (Docker’s “host networking”), so two Avahi servers ran on one machine and both claimed nas.local. The container could also see the NAS’s internal Docker networks, so its answer listed two extra addresses. Different answers for the same name meant each server treated the other as an impostor, around the clock. My NAS’s server got as far as -319. The container’s got to -67.

“Neither server was broken. Each one followed the rules exactly, and that is what kept the fight going.”What I took away from case No. 002
Exhibit C · The rename loop
DSM’s Avahi sees eth0 .5, eth1 .6 Container’s Avahi sees eth0, eth1 + Docker bridges loop · 2 weeks ① probe: anyone own nas.local? ② yes: A 10.0.0.5, 10.0.0.6 ③ my set: .5 .6 172.17.0.1 172.26.0.1 different → conflict → rename to -2 ④ announce nas-2.local with 4 addresses ⑤ rival claim on my IPs → rename ended as nas-319 ended as nas-67
Two honest daemons, one name, different answers. Neither was broken; each was following the probing rule correctly. The printer advert ended up pointing at a host name that nobody was publishing an address for, so your iPhone could list the printer but could not connect to it. It also explains why nas.local stopped resolving from your Mac.
Exhibit D · how I fixed it

One owner per name

The fix was a rule both servers could live with. My NAS’s Avahi owns the NAS name and its addresses. The container’s Avahi stops publishing a name or addresses at all and only announces the printer. Its SRV record still points at nas.local, which the NAS answers for. If you run Avahi in a Docker container with host networking, these are the three lines to copy into the container’s avahi-daemon.conf:

[publish]
publish-addresses=no
publish-hinfo=no
publish-workstation=no
Exhibit D · Before and after the fix
Before nas · one network stack DSM’s Avahi A nas .5 .6 Container’s Avahi A nas ×4 incl. 2 docker IPs PTR printer SRV → nas TXT capabilities ! same name, different records: endless renames After nas · one network stack DSM’s Avahi A nas .5 .6 only Container’s Avahi publishes no name or addresses PTR printer SRV → nas TXT capabilities printer records point at DSM’s name: no conflict
The difference is one record. Only the container’s A record moves: it disappears, and the SRV record that used to point at the contested name now points at the one DSM answers for.
Exhibit E · the whole route

From Share → Print to paper

Bonjour only gets the iPhone to the NAS. After that, printing is ordinary networking. Two of my other fixes sit on this route: the NAS sends Epson’s native ESC/P-R format to the printer’s raw port, and it no longer uses the backend that kept crashing.

Exhibit E · The print path
iPhone Share → Print CUPS on the NAS nas.local:631 epson-escpr renders ESC/P-R Epson L3250 10.0.0.7 IPP, PDF filter TCP 9100 Bonjour: PTR → SRV → TXT → A ipp backend · crashed (signal 11) not used any more: it crashed on every job inside the Alpine container
Bonjour is only the first hop. The dashed blue path happens once, when the print menu opens. The solid path carries the actual job.
Obituary

The ipp backend, CUPS 2.4.19

Died 19 October inside an Alpine container, on every job it was given, of signal 11. It is remembered for crashing just after checking each job was valid, while its printer-monitor thread was running. It is survived by a plain socket to port 9100, which now carries every page. In lieu of flowers, please send ESC/P-R.

The photo came out at 21:03. Fifteen minutes later my NAS had its own name back: nas.local, no number. One name, one owner. And the printer, the first suspect anyone looks at, had been innocent all along.

Case closed
If this happens to you

A printer that shows up but won’t print: check these in order

  1. Is the queue paused?

    On the print server, a queue marked disabled has given up. Turn it back on, and set it to retry so one failure can’t stop it for months.

    lpstat -p
    cupsenable <queue>
    lpadmin -p <queue> -o printer-error-policy=retry-job
  2. Did the printer’s address change?

    Give the printer a DHCP reservation in your router, and point the queue at that address.

  3. Does the host in the advert resolve?

    The first command shows which host and port the printer’s advert names; the second must return an address for that host. If it doesn’t, your phone can list the printer but never reach it.

    dns-sd -L "<printer name>" _ipp._tcp
    dns-sd -G v4 <host>.local
  4. Is something renaming itself?

    A name ending in -2, -3 or higher means a conflict. On a Linux box or NAS, check what Avahi is calling itself:

    ps -eo args | grep "[a]vahi-daemon:"
  5. Two Bonjour servers on one machine?

    With Docker host networking, let the host own the name. In the container’s avahi-daemon.conf, set publish-addresses=no, publish-hinfo=no and publish-workstation=no.

  6. Gibberish from port 9100?

    The raw port expects the printer’s own language. Use the manufacturer’s driver (ESC/P-R for Epson inkjets) or send the job over IPP instead.

  7. “Backend failed”?

    One of CUPS’s helpers crashed. Its error_log says which one and how, so read it before blaming the printer.

Key takeaways
  1. Give every network name exactly one owner. Two honest servers with different answers will fight forever.
  2. When a fix almost works, look for the next layer. Each of my five problems hid the one underneath it.
  3. Silence isn’t health. Make things retry and complain: my queue stopped in June and never said a word.
Field guide

Where else this can bite you

It stays on one network

Multicast does not cross routers, VLANs or subnets. A guest Wi-Fi on its own subnet will not see the printer unless the router reflects mDNS.

Answers are cached

Devices remember adverts until their time-to-live runs out, often 75 minutes. That is why a stale “@ nas-67” lingered after the fix.

Wi-Fi client isolation

Some access points stop wireless devices talking to each other. The question never reaches the printer.

Port 5353 must be free

A firewall blocking UDP 5353 breaks discovery with no error. Two servers sharing that port on one machine is how this whole story started.

Try it at home

Explore Bonjour on your own network

Each command runs until you press Ctrl-C. Work down the record chain from the top, the way Exhibit B does.

List printers (PTR records)Mac Terminal
dns-sd -B _ipp._tcp
One printer’s host, port and capabilities (SRV + TXT)Mac Terminal
dns-sd -L "AirPrint EPSON_L3250 @ nas" _ipp._tcp
The NAS address (A record)Mac Terminal
dns-sd -G v4 nas.local
Every kind of service on the networkMac Terminal
dns-sd -B _services._dns-sd._udp
Is the NAS renaming itself? It should say nas.local with no numberNAS over SSH
ps -eo args | grep "[a]vahi-daemon:"
Puzzle corner

Test yourself

Q1Your iPhone lists a printer but every job fails to connect. Which record in the chain is most likely missing?

The A record for the host named in the SRV record. The PTR, SRV and TXT records got it into the list; without an address the phone has nowhere to send the job.

Q2Two Avahi servers on one machine both announce nas.local. Why is that only sometimes a problem?

Identical answers are allowed. It becomes a conflict when the records differ, as here, where one server also listed the Docker bridge addresses. Different answers trigger the rename rule.

Q3Why did printing raw data to port 9100 produce gibberish?

Port 9100 passes bytes straight to the printer. The data was PWG raster, which this Epson only understands over IPP, not on its raw port. Sending ESC/P-R, the printer’s native format, fixed it.

Glossary

Words from this issue

mDNS
Multicast DNS. Name lookups answered by the devices themselves on the local network.
DNS-SD
DNS service discovery. Uses PTR, SRV and TXT records to describe services like printers.
Avahi
The Linux implementation of mDNS and DNS-SD. Synology DSM uses it.
Probe
The “anyone own this name?” question a device asks before claiming a name.
Host networking
A Docker mode where the container shares the host’s network stack, ports included.
IPP
Internet Printing Protocol. How the iPhone sends the job to CUPS, on port 631.
ESC/P-R
Epson’s native printer language. What the printer actually understands on port 9100.
URF
Apple’s raster format for AirPrint, and the TXT key that advertises support for it.