unixdude.net

PSR-100 satellite rotor

2026-02-10T00:00:00-05:00

I have long been fascinated by ham radio satelites, but once you get past the most basic setup of an Arrow II or Elk antenna and an HT, the required equipment can become prohibitive in both size and expense.

Enter the PSR-100, a small rotor kit that is designed to hold those handheld Arrow II and Elk antennas. This is both cheaper and smaller than the amazing Yaesu G-5500/M2 LEO Satellite antenna setup. I am sure it will be less performant, but right now for me size and cost are top concerns.

I really like the Yaesu G-5500 setup with the M2 LEO Satellite antennas, but I live in an HOA where that type of setup is not allowed, so I started looking for something that would effectively be the little brother of the awesome 5500/M2 setup. This PSR-100 + an Elk or Arrow II antenna seems to be just that.

I got my kit last week and soldered the board over the weekend. I should finish my build this week, then I will be able to try it out.

This kit requires a Windows laptop to control the rotor. It has an interesting design where the link between the computer and the rotor is not wired but is instead a private wifi network between the kit's two ESP32 devices. One ESP32 is installed on the rotor control board and the other is found in a USB dongle that plugs into the computer. The companion Windows software uses the private wifi connection to control the rotor.

The Windows software can also act as a go-between to allow the rotor to be controlled by a CSN SAT controller, with the CSN SAT configured to control a PST Rotator, using the laptop's IP address as the rotator IP address.

My plan is to use this rotor with an Elk antenna, a CSN SAT controller, and an Icom IC-9700.

I chose the Elk over an Arrow mainly because I wanted a faster and simpler setup. Since I cannot have a permanent satellite station at home, I will be using this in portable/temporary setups, both in my driveway and in parks, so setup time and complexity must be considered. There are trade-offs with this setup: Instead of having 2 cables, I will instead need a duplexer, but I already have a Comet CF-416B, so I will be using that. I have ordered a coax cable from ABR Industries, and that should arrive soon.

Based on what I have seen so far, this kit is top shelf. This kit came extremely well-packed and ready to build. I didn't take a picture of it when I opened the box, but I took pictures of the box and then all the components. Don has done a great job with this kit, as you can see.

Don has a comprehensive pre-assembly test guide, and I took this pic while I was running those tests.

I look forward to finishing this project and then using this setup to make satellite contacts. I might even try to work satellites during a POTA activation.

Comics

2025-10-26T00:00:00-04:00

For 15-20 years, I have run a script that gathers daily images for several comic strips. Recently, this script stopped working, and rather than fix it (it is Perl code that at this point has become unmaintainable) I decided to look for an OSS solution.

Web searches quickly led me to dosage.

Dosage can output HTML files, and it seems that, generally, people have Dosage output saved to a directory accessible on a web server. I wanted to do this too, but I also wanted to have this fully deployed via Ansible, and I wanted it to integrate with my Traefik server, with everything running in Docker images.

Also, there is an issue affecting Dosage's ability to download from GoComics, but there is also a workaround (posted in that thread) that is not yet available in the main Dosage code, so I wanted my Docker container to incorporate that fix.

I ended up with a very simple Docker container, a volume to store the images, and a cron job to run load the strips each day.

Credit for the download.sh script goes to Simon Szustkowski, in his dosage-docker repository.

I tried out Simon's image, but found that it was not suitable for my needs, due to the GoComics issue mentioned above. And, my solution is not a fork of his, but rather a similar solution.

The code to build my Dosage docker image is found here.

I'm using this as a one-shot container -- I do not leave it running.

The other side of this is having it on the web. Since I am already running Traefik, and since I only need to present a few files on the web, I decided to add this as another web server container connected to Traefik. Among other benefits, this means I easily get SSL using my existing configuration.

All I had to do on this side was to add another DNS entry (comics.unixdude.net), another call to my traefik-website role, and a little bit of configuration of that new role.

The only remaining pieces were to add a directory in which to store the images, and a cron job to run the dosage-docker container once daily.

New NAS

2025-08-24T00:00:00-04:00

I recently discovered that my offsite backup NAS, a Synology DS118 with a 12TB drive in it, had filled up. I considered several solutions to this problem, eventually deciding to replace the DS118 with a DS225+. Since Synology now requires Synology-branded drives, I had to either buy two new drives or I had to shift some drives around.

To save some money, I decided to shift drives around rather than buy two new drives for the DS225+, because the 12TB drive would work nicely in my local backup NAS, and I recently purchased two 6TB Synology HDDs to replace dead drives in my local NAS units. I decided to buy a single 16TB Synology drive in conjuntion with using one of my recently purchased 6TB drives; as a result, I have about 22TB in the DS225+.

I decided to configure these drives as two basic volumes, rather than as a single volume, because I don't care about redundancy for this unit, and because I do not want a single failed drive to destroy all of my remote backup data, which is what would happen in a JBOD setup.

After I installed the 16TB drive, I started priming the backup. For my remote backup solution, I use Synology's Hyper Backup because it works well, works fine over WireGuard, and is easy to use. A short while after starting to prime the new backup, I noticed that the operation was taking forever... I traced this down to a bad switch port on my desktop switch: The backup was running at only 100Mbps! Simply relocating the cable fixed the issue.

Once the backup is primed, I will take it back to my friend's house, and I will reconfigure Hyper Backup to point to the new location.

Access to the unit will be over WireGuard -- the DS225+ will connect to one of my DigitalOcean VMs, and OSPF will allow me to easily route to it. For WireGuard on the Synology, I used this guide to find the installation SPK, then I followed the standard WireGuard setup. I generated the key by running something like this: wg genkey | tee private.key | wg pubkey > public.key.

Then I configured the WireGuard client on the DS225+ by creating /etc/wireguard/wg0.conf with these contents:

[Interface]
Address = 192.168.x.x/30
SaveConfig = false
Table = off
PrivateKey = GENERATED_PRIVATE_KEY

[Peer]
PublicKey = PEER_PUBLIC_KEY
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = hostname:port
PersistentKeepalive = 3

Once the peer was configured as well, a simple wg-quick up wg0 on the DS225+ was all I needed to establish the connection.

In order to ensure that this connection is started when the NAS boots, I also ran wg-autostart enable wg0 on the DS225+.

WireGuard + FRR really is a great solution for remote systems like this offsite backup NAS.

Ultimate Hacking Keyboard

2025-05-11T00:00:00-04:00

I recently received some advice that I should set up a more ergonomic workstation at home: a split keyboard, an ergonomic mouse, and an adjustable sitting/standing desk.

I did a lot of research about keyboards and decided that an ortholinear or column staggered unit was not for me. I wanted something that would work in both split and one-piece configurations, and something somewhat close to what I already knew because I do frequently use the onboard keyboards on my laptops. In my research, I eventually discovered the Ultimate Hacking Keyboard. This is a fully programmable unit, with layers and macro support.

Right around the time I was looking for a new keyboard, a fellow employee in California listed his spare UHK60 v2 for sale, including a carry case and both the 3-key cluster and trackpoint modules, and I bought it.

I received it on Monday of this week, and I immediately plugged it in and started using it. I had never used a programmable keyboard before, and I had never used a split keyboard, and it has been 20 years or more since I last used a mechanical keyboard.

My previous keyboard was the Apple one -- I have a 2025 MacBook Air, a 2024 MacBook Pro, and an A1314 -- all have the same basic Apple layout (the newer ones have fingerprint sensors in the location where the old one had an eject key; this is the main difference). The UHK60 is a 60% keyboard, so when compared to the Apple keyboards I have used for decades, the UHK is missing the function keys, escape key, and arrow keys. All of these are accessed through layers on the UHK, a layer meaning that you press a modifier key (such as the UHK's Fn or Mod key) to access that key.

My first day was very rough -- I kept my A1314 close at hand, just behind the UHK, and I kept using the A1314 to do things that I had not sorted out how to do on the UHK. It was easy to get used to the split keyboard; it was difficult to get used to accessing keys through layers.

On Tuesday, the A1314 was a little farther away to the side, but I still used it quite frequently. By this point I had started reconfiguring the UHK to have it do what I wanted it to do... somewhat like how I decide where to put things in my house: Where will I look for that thing? When I found myself doing a particular key sequence for a function, I made that shortcut do that function, rather than always forcing myself to learn the default UHK shortcut for that function. This is, after all, the point of the UHK: you program it how you want it to operate.

The default UHK layout for the modifier keys matches that of a PC keyboard, not a Mac keyboard, so one other thing I did early on was to pull all of the modifier keycaps and reorder them to be more like the Apple keyboard, but I swapped the left function and left control keys, because the order of those two keys has always annoyed me on the Apple keyboard. Doing all that resulted in a much higher positive hit rate, and on Wednesday my old A1314 keyboard was at the edge of my desk, still powered on and available if I needed it, but I hardly used it. On Thursday the Apple keyboard was in the drawer, powered off.

There is much to learn here, and I am almost certainly going to change out the silent pink switches in this keyboard for something with at least tactile feedback, but most likely I will go with a clicky key. I am going to test out a few switches and then buy replacements.

At this point I greatly prefer the UHK over the Apple keyboard, and look forward to learning everything that it can do, and further tweaking it to best fit my usage.

As you can see in the first picture, I have also replaced my Apple Magic Mouse with a more ergonomic Logitech MX Master 3S. So far so good with that one.

Bye bye Time Machine

2024-08-24T00:00:00-04:00

In my family, we have a half-dozen Mac laptops -- one for each of us, and one provided by my employer. For the personal laptops, I have been using Time Machine, backing up to my primary Synology unit. Unfortunately, Time Machine has proven unreliable, requiring frequent resets of the backup. For every system, I have had to reset the backups multiple times -- that is, I have had to completely erase the backup on the Synology, and have Time Machine do another full backup. This happened to my laptop backup so many times that a few months back, I migrated my laptop's backup to rsync, using an NFS automount on my Synology as the destination. I run this in a cron job, and I use healthchecks.io to watch the status of the cron jobs.

The cron job is simple, just something to run the script every 2 hours:

0 */2 * * * root /Users/admin/bin/rsync_mba

For the health check, I use a simple check type, with a 1-day period and a 2-day grace time. I use the 2-day grace time because I do not want to get notified of a failed backup just because I unplugged my laptop for the weekend.

Today, I configured this style of backup for all of my family laptops. I chose to do it now because my younger son's laptop refuses to restart Time Machine: it repeatedly warns that the TM hard drive was replaced, and forces me to confirm that I want to use the drive -- but then it never starts the backup. So, his system has been without a backup, until today when I configured this rsync style backup.

The script itself is also simple, basically just an rsync, but with a bunch of excludes for things that need to be excluded on macOS. Also, the script maintains status files and it has a variable holding the healthchecks.io ping URL.

The script checks the exit code from rsync, and does various things based on the exit code. For example, rsync exit code 23 means the transfer was partially successful, and I log this as a success, but I also note that it happened. rsync exit code 24 means that some source files disappeared. I log this as a success without qualification.

One thing I maintain is a travel router; this is new in my toolkit, and I should post about it. Anyway, a short summary is that my travel router has an always-on WireGuard VPN to my home network via one of my Digital Ocean VMs. I do not want backups to occur while I am connected this way, so my NAS sets the export to be read/write only for my home network's IP subnet. This way, I do not eat up lots of bandwidth while I am not at home -- such as when I am using Starlink or wifi at a condo or resort.

F150 radio install update

2024-05-26T00:00:00-04:00

A while back, I moved my FT-7800R to my hamshack, and bought an FTM-400XDM to replace it in the truck, so I thought an update was in order. I never fully mounted the FT-7800R control head, but after switching to the FTM-400, I finished the permanent install by mounting the control head.

I bought a Bulletpoint Mounting Solutions RubiGrid for my F150, and some RAM Mount balls and arms to mount the control head.

Because the FTM-400 microphone only connects to the main radio unit, I needed to get an extension cable for that. I also needed a longer cable for the control head, which I built myself since it only needed flat 4-conductor cable and RJ 4P4C connectors, which I already had on hand.

I'm pretty impressed with the results of this install.

Home lab

2024-05-26T00:00:00-04:00

I was recently asked about how to get started with a home lab. As with most things, the answer depends on what you want to do with it. But, the core components are probably pretty similar.

Every home lab is going to require some sort of storage system. For mine, I use Synology units running Btrfs. I currently run a DS1821+ as my primary storage unit, with 7 disks in an SHR-2 configuration, and one SSD cache disk. I chose SHR-2 for its RAID6-like double disk redundancy. My disks are so large that the likelihood of a second failure during a rebuild was high enough that I wanted to protect against that.

I have a second Synology at home, a DS1618+, for backup purposes. My first Synology sends Btrfs snapshots to the second. This backup unit uses SHR-2 like the primary, but with only 5 disks in the array.

Most home labs will implement some sort of virtualization. For my needs, free VMware ESXi on an i7-powered Lenovo M900 with 32GB RAM works well. I have tried Proxmox, and I greatly prefer ESXi. I know there are other solutions, and I will need to evaluate those since Broadcom has done away with the free license, meaning home lab users are out of luck with that one now. When I had a Dell R610 with 192GB RAM and was a member of VMUG, I ran vSphere, but that takes too many resources on my M900.

I chose the M900 because at the time of my purchase, they were readily available used on eBay for a couple hundred dollars, and the i7 model supports 8 vCPUs and 32GB RAM. Any of this type of tiny server (there are many) should work. My main recommendation is to get an i7 and something that supports at least 32GB RAM.

For my network, I exclusively use Ubiquiti products. I have multiple sites and multiple VLANs. I run OSPF and WireGuard to connect those sites. This is more than a beginner will need to do.

Other considerations:

Backup power; I use a CyberPower 1500AVR.
Monitoring; I run Kuma for basic monitoring

For additional reading, I recommend Hayden James's excellent guide.

Home network upgrade

2024-04-21T00:00:00-04:00

Over the weekend, I upgraded my network: I replaced my Ubiquiti USG with a UDM-SE. I debated between the UDMP and UDM-SE, and decided on the UDM-SE because of its added features. As part of this upgrade, I retired my self-hosted UniFi Controller in favor of the one on the UDM-SE.

I have never done a UniFi network upgrade before, so I really wasn't sure what to expect. It was actually surprisingly easy, but my custom setup required a slightly modified process and configuration after the upgrade.

My biggest questions were: How does one actually replace a UniFi router? How does one replace the router and controller with a single unit, as I was doing? I did some research but was never truly clear about the answers to these questions.

Replacing a controller seems to be a pretty well-known process: Take a backup, then forget all devices in the old controller, and import the backup into the new controller. My concern was that I was also replacing the router, and I did not know what the controller in the UDM-SE would do with the USG in the backup. It did exactly what I wanted it to do: It detected the USG in the backup, and it replaced the USG with itself.

I thought I was going to have to adopt the UDM-SE and somehow figure out how to replace the USG with it, because I thought I would need to configure the UDM-SE with an IP address to add it to the network, then replace that address with the one in the USG, but I was completely wrong.

I had to use a slightly different process because of the configuration I had on the USG: I used a config.gateway.json file to configure WireGuard and OSFP, and I had to remove that before taking my backup. After the migration, I had to add my WireGuard links and configure OSPF on the UDM-SE.

In the end, this worked great except for one thing: The web UI does not seem to allow me to configure OSPF on the WireGuard links. OSPF on the UDM-SE peered with my local routers, but not with my remote ones, so my routes were not optimal. There are multiple ways to get to my remote sites, so the routes were there, they just added another hop or 2. This kind of defeats the purpose of WireGuard for me.

I did a lot of searches, and found this 7-month-old post, and wondered if it would still work in the current version of the OS. Turns out it works great: After starting the FRR service, I was able to use vtysh to add the WireGuard links to my OSPF area, and to add the neighbors. It required one more thing to work. I had to configure the interfaces with ip ospf network non-broadcast:

interface wgclt1
 ip ospf cost 1
 ip ospf dead-interval 40
 ip ospf network non-broadcast
exit

These entries mirror the other ones created on the UDM-SE, and the ip ospf network non-broadcast mirrors what I had on the USG.

After I did all that, OSPF peered to the routers at the remote sites, and the correct entries were in the routing table. The config persists across reboots, and it works perfectly.

Solenoid installs

2024-01-28T00:00:00-05:00

A few weeks ago, I installed a solenoid in my truck, to control my ham radio's power based on the ignition key position: I want the radio to power on and off with the engine.

As with most mobile ham radios, my radio has an auto-power-down function, but I want it to power on as well. At the suggestion of a friend, I purchased these solenoids. I don't know if all ham radios do this, but mine power up when they regain power, if they were powered on when they lost power. So, the solenoid will power my radio up and down.

This solenoid is easy to use, as can be seen by the wiring diagram:

The hardest part of this type of job is getting through the firewall. Fortunately, I had already done that when I wired up my FT-7800R in my truck. Installing this solenoid involved only some rewiring, and placement of the solenoid.

First, we had to find a switched trigger source. We searched the fuse box and found an unused pin that was powered only when the ignition was switched on, so we used that, wiring it to the "positive trigger source" pin on the solenoid.

The radio ground was unchanged. The radio power source was changed from battery to solenoid, and the solenoid is powered directly from the battery. It's a pretty simple install, as you can see.

Yesterday I added a solenoid to my wife's 2010 Lincoln Navigator. I did this because we now have a third driver in the family, and I wanted a dashcam to have some video in case we need it.

At a friend's suggestion, we bought this dashcam. I'm not yet going to recommend this camera, but it seems okay so far. In any case, yesterday I finished the install, with the same professional-grade install that I did for the radio.

First, we had to find a switched trigger source. We were able to locate an unused (and powered!) fuse slot in the fuse box, so we used this adapter to make our own circuit. This goes to the solenoid positive trigger pin. Note that we also get the positive power line from the fuse box too.

We located the solenoid on the firewall at the top of the engine compartment, similar in location to where we did it on the F-150.

As with the F-150, the hardest part of this job was pulling cables through the firewall. (The next hardest part was pulling them behind the A-pillar trim.)

This camera came with a 12v power plug, and I didn't want to cut that off, so I bought a 12v socket. I then soldered that to the wires we passed through the firewall, and I plugged the camera into this. Then I zip-tied this to a U-shaped beam under the dash.

All of the wires are hidden behind trim of course. Camera is suction-cup mounted to the windshield, and now it powers on with the ignition key.

ESPHome home control

2024-01-15T00:00:00-05:00

I have continued playing with my M5stack. I now have it programmed to show 3 types of data:

The local weather forecast for the next several days; either the 5-day forecast, or a single day at a time.
The temperature and humidity readings in my home office and garage, or the reading from a single location.
The status of my Kasa Smart outlets. This view also controls some of the outlets.

I already posted about the weather forecast. This post is about the other 2 items.

My ESP32 in the garage has a PIR sensor as well as a DHT11. The PIR sensor is used to detect motion, which then initiates an automation to turn on the garage light. This has been super helpful, because we do not have to leave the garage light on: it always turns on when we enter the garage. The DHT11 provides temperature and humidity readings in the garage.

One of the reasons I wanted the M5stack was to be able to use it as a remote control for the house, and I have now achieved that. The code for this is a mess, but it can be found here.

I would prefer to use arrays for things like the names of the switches (both my "friendly name" and the Home Assistant name), and I would prefer to use array length to know when the last item is being viewed. I assume it is possible to do this in ESPHome, but I don't know how to do it yet.

In any case, I can now cycle through the lights, and I can control all but the garage light (because there's no need to control this one manually). This functionality was suprisingly easy to add tonight, and it is super cool: Now I can carry my M5stack, and push a few buttons to control the lights around the house.

ESPHome weather display on an m5stack

2023-12-10T00:00:00-05:00

After using a couple of ESP32s and doing some things with them, I decided I wanted to get an M5stack. I bought a first generation core, and am working to use that as a display for various things around the house.

As I started using the m5stack and trying to do this project, I did not find a complete ESPHome m5stack "hello world" type program, or a complete ESPHome weather "hello world" type program, so in this project and posting, I create both of those. I uploaded the ESPHome code to my github repository.

Note from my code that I am using font Roboto-Medium.ttf. You can use any font you like, just make sure it is uploaded to your device.

Also, hopefully others find this as useful as I intend it to be, and as useful as I would have found something like this, when I started on this project.

I decided my first project would be a weather forecast display. This is a project that many people have done, and tonight I was able to get the data to my M5stack.

As I said, I consider this a "Hello World" type program, and I post it here because it took me a while to piece this all together. I found lots of writeups from people who had done this project, but they seemed to be assuming knowledge that I didn't have; hopefully this post includes everything a new user needs to get this done.

First, you need to create a sensor in Home Assistant. To do this, edit your /homeassistant/configuration.yaml file. I added the following code to mine, using the Home Assistant File editor. If you do not have the File editor installed, you can add it as an Add-on.

sensor:
  - platform: template
    sensors:
      wx_forecast:
        friendly_name: "Local forecast"
        value_template: >-
            {% set weather = {
               "sunny": "sunny",
               "clear-day": "clear-day", 
               "clear-night": "clear-night", 
               "cloudy": "cloudy", 
               "rainy": "rainy", 
               "sleet": "sleet", 
               "snow": "snow", 
               "wind": "wind", 
               "fog": "fog", 
               "partlycloudy": "partlycloudy", 
             } %}

             {% for state in states.weather.forecast_home.attributes.forecast[1:6] -%}

             {{ as_timestamp(state.datetime)| timestamp_custom("%a") }};{{state.templow}};{{ state.temperature }};{{ state.precipitation }}in;{{ weather[state.condition] }}#

             {%- endfor %}

This code is basically copied from this post, but I removed the day name translation. I could also remove the weather hash, but I kept that for now. I also changed the format slightly - I separate the low and high temperatures into separate fields.

After I edited the configuration.yaml file, I checked and then reloaded the configuration on the Developer Tools page, YAML tab. After the reload, I confirmed the existence of my state, by going to the States tab on the Developer Tools page, and filtering states for wx_forecast. This completes and confirms the HA side.

Then, in ESPHome, I added the sensor to my m5stack configuration:

text_sensor:
  - platform: homeassistant
    id: w_forecast
    entity_id: sensor.wx_forecast

My display code is basically copied from the same post as above. Here is my display lambda code:

  std::string fivedays = id(w_forecast).state;
  std::vector<std::string> five;
  std::vector<std::string> v;

  ESP_LOGD("fivedays [%s]", fivedays.c_str());
  five.clear();

  int count = 0;
  int wx = 0; // start position x
  int wy = 0; // start position y
  char *token = strtok(const_cast<char*>(fivedays.c_str()), "#");
  // this while splits the string (I believe, I "found" the code)
  while (token != NULL)
    {
      five.push_back(token);
      token = strtok (NULL, "#");
    }
    // here we loop the days
    for ( std::string fiv : five ) {
      //it.rectangle(0, wy, 128, 59);  // adds a border around the "day"
      //it.rectangle(1, wy+1, 126, 57);

      std::string str = "";
      str = fiv;
      ESP_LOGD("test: ", "String to Vector: %s", str.c_str());
      v.clear();
      token = strtok (&str[0],";");
      while (token != NULL)
      {
        v.push_back(token);
        token = strtok (NULL, ";");
      }
      // this is the loop for each value in the "day"
      for ( std::string s : v ) {
        if(count == 0){
          // Day (Mon/Tue...)
          it.printf(wx, wy, id(font_roboto_medium22), id(my_red), TextAlign::TOP_LEFT, "%s", s.c_str());
        }else if(count == 1){
          // Temperature
          it.printf(wx +60, wy, id(font_roboto_medium22), id(my_red), TextAlign::TOP_LEFT, "%s", s.c_str());
        }else if(count == 2){
          // Precipitation
          it.printf(wx +100, wy, id(font_roboto_medium22), id(my_red), TextAlign::TOP_LEFT, "%s", s.c_str());
        }else if(count == 3){
          // weather icon
          it.printf(wx + 135, wy, id(font_roboto_medium22), id(my_red), TextAlign::TOP_LEFT, "%s", s.c_str());
        }else if(count == 4){
          // weather icon
          it.printf(wx + 210, wy, id(font_roboto_medium22), id(my_red), TextAlign::TOP_LEFT, "%s", s.c_str());
        }

        ESP_LOGD("test: ", "String to Vector: %s", s.c_str());
        count += 1;
      }
      count = 0;
      wy += 25; // move down 25 pixels and output next day
    }

The result of all this work is a weather forecast on my m5stack:

Now that I have the data in my m5stack, I will set out to get the display how I want it.

Home Automation using Home Assistant

2023-11-28T00:00:00-05:00

A few weeks back I mentioned Home Assistant. After I set that up, I bought some Kasa smart outlets. Two have replaced timers, so I no longer have to change the start times as the time of sunset changes throughout the year. A third is set up in my home office, and is set to come on an hour before sunset. I installed this plug because I often find myself in a dark office at the end of my work day. I don't even notice it getting dark outside because I'm staring at lighted screens, so this keeps me out of the darkness when I shut down.

A couple days ago, I installed the 4th plug. This one is in the garage, and it now switches one of the two LED shop lights that I installed on Saturday. The other LED shop light is switched inside the house. Originally, both of those LED shop lights were switched inside the house, but that is two 5,000 lumen shop lights -- and those replaced two 825-lumen compact fluorescent bulbs -- so it was extremely bright, much brighter than we were used to. I moved one of the shop lights to the Kasa switched outlet, which cuts down on the brightness, but means that I would have to manually power the second light, from the app on my phone.

I happened to have a spare ESP32 and a HC-SR501 motion sensor. A quick web search revealed this tutorial, which I used to get started. I set my sensor for maximum time delay as well as maximum range. I configured the sensor in Home Assistant, then, added two automations for the sensor: turn on the power to the garage light smart plug when motion is detected, and turn off the power when motion is no longer reported. With the long time delay, we get several minutes of light before it turns off.

This entire setup and automation took me maybe 30 minutes, probably much less, and this is one reason why I like ESPHome: It is easy to automate with an ESP32 and a sensor. The entire config is in this one stanza:

binary_sensor:
  - platform: gpio
    pin: 13
    name: "motion_sensor"
    device_class: motion

The automations are super simple too: I set up a trigger of "garage_sensor motion_sensor started detecting motion," and the action is to turn on the outlet. I have a related automation with a trigger of "garage_sensor motion_sensor stopped detecting motion."

ESP32s are so inexpensive, too, which makes automation like this super affordable. Now I'm trying to decide what other automation projects I will do. I think I will automate garage door open detection, as well as closing it. I also want to automate the lights on the 2nd/3rd floor staircase: the stairs are very dark. I'm sure I will come up with other ideas as well.

Home Assistant; Ham radio digital voice

2023-11-05T00:00:00-04:00

A while ago, I bought a Freenove starter kit. I never did much with it until recently, when a friend gave me an ESP32 and pointed me to Home Assistant.

The Freenove kit included a DHT11 temperature/humidity module, so my first Home Assistant project was to set up a temperature/humidity sensor for my home office. It was much easier than I thought it would be: I downloaded and installed the Home Assistant OVA, then I installed ESPHome on the ESP32, then I configured the ESP32 with the DHT11 device.

This is inexpensive, useful, and cool. I can see me buying many of these ESP32 or similar devices, using them to monitor and control various things around the house. And, with the WireGuard support in ESPHome, I will be able to monitor remote devices as well, such as the temperature in my travel trailer, which is parked at a friend's house.

Another thing I have been playing with recently is digital voice in ham radio. A friend lent me a Yaesu FT-70D and a MMDVM hotspot running WPSD. After attaching the hotspot to my home WLAN, and a small bit of configuration, I have the FT-70D connected to the hotspot, and have a world of digital voice radio available to me, through the many YSF reflectors that exist.

In addition to the ESP32 modules and sensors mentioned above, the FT-70D and a MMDVM hotspot are now on my list.

IC-7300 external keypad

2023-10-03T00:00:00-04:00

I recently purchased an Icom IC-7300 ham radio for use at home, and when researching the radio around the time of purchase, I discovered this excellent YouTube video playlist, covering many aspects of the radio.

When I got to video 46, I discovered that the IC-7300 supports an external keypad, and decided I wanted one of my own. This past weekend, I made one.

In addition to using that video, I found this page detailing another guy's external keypad build.

I don't have a drill press, but I think my unit came out well. I could have done a better job with the feet, but this works well enough.

Mobile Wireguard clients

2023-07-02T00:00:00-04:00

I have been using Wireguard for about a year & a half, when I switched from ZeroTier to Wireguard for most of my needs. Unfortunately, since the start, my Wireguard configuration has been completely manual, and very annoying since I had to create a new interface for every client.

No more.

Today I used the Wireguard Config Generator to generate the configs (I did not specify my endpoint on the website of course). The result was a single server config with keys for two dozen mobile clients. Today I configured three of those mobile clients -- one each for my MacBook Air, my iPhone, and my iPad.

I considered using Netmaker but decided it was way more than I needed. The config generator made that part of the config easy. Loading the config into the Mac client was easy. Loading a config file into the iOS client is not difficult, but it was way more fun using qrencode to generate a QR code to configure the iOS devices.

After I used the Wireguard Config Generator, I updated the client files to specify my endpoint, then I loaded the configs and was off to the races -- with that part anyway.

I had one more step to make this useful. The entire point of my Wiregaurd config is to remotely access my home network. But, since my Wireguard server is on a DigialOcean VM, and I want my systems to access my home network without masquerading, this required a configuration change to my FRR config on my DigitalOcean VM.

I just had to add two blocks:

interface wg0
  ip ospf network non-broadcast
exit

To the router ospf configuration, I added the subnet I dedicated to my Wireguard clients:

network 192.168.100.128/27 area 0

Now everything can route to my Wireguard clients, and my Wireguard clients can access my entire network.

POTA kit

2023-06-14T00:00:00-04:00

Parks On The Air, or POTA, is one of the aspects of amateur radio that I have been enjoying -- both hunting and activating. Ever since I starting doing POTA activations, I have been carrying my equipment in a Rubbermaid container, which is bulky and heavy. The only thing it has going for it is that it carries everything I need.

Pretty much since getting started doing POTA, I have wanted a battery box and a radio box. I did a lot of research about battery boxes, and there are many different options, both DIY and ready-made ones for purchase. Somewhere I saw a build that used the Harbor Freight Tactical Ammo Box, and I decided to use that box for my build. I went with this box because I really like the idea of having all of the controls and connections under the flip-top lid on that box, rather than on the outside of the box, which is how most of these boxes are built.

Over the weekend, with the help of a local ham friend, I built my battery box. Installed on my box are dual PowerPole connectors, dual USB connectors behind a rocker switch and 5A fuse, and a 50A circuit breaker as the main power switch. There is room for more connections should I choose to add them.

The Harbor Freight box I used has the benefit of being a little larger than the typical battery box, so in addition to the Bioenno BLF-1215A, I am also able to carry my 40/20/15/10 EFHW antenna, NanoVNA, battery monitor, and Bioenno battery charger -- all inside the battery box. Today, the ability to carry all these extra items seems like a benefit; time will tell if that impression holds.

I also finally cut out the foam in my Apache 3800 case to carry my radio (an Icom IC-706MKIIG), along with its microphone and power cord, and nifty mini-manual.

I am very pleased with the result for both of these, and look forward to using this kit at an upcoming POTA activation.

DKIM, Postfix, and FreeBSD

2023-05-10T00:00:00-04:00

Recently, I discovered that I was not able to email my wife's mac.com address from my mail server here. Apple's support document indicates that they require DKIM and SPF, and that they support DMARC. I have long had SPF configured, but I had not configured DKIM or DMARC.

Over the weekend, I set up DKIM. DKIM is basically a public-key cryptographic solution used to confirm the server that sent an email. In order to set this up on my FreeBSD 12 server, I followed this guide which is pretty close to what I ended up doing, and I took some inspiration from this guide. Even with these guides, I had to make a few changes to meet with success on my FreeBSD 12 system.

I chose to install the package, not using source: pkg install opendkim

Next, instead of adding the user to the mail group, I left off that parameter:

pw useradd -n opendkim -d /var/db/opendkim -m -s "/usr/sbin/nologin" -w no

I had to do this because I was getting errors while trying to start opendkim. The errors complained that the opendkim user was a member of the mail group, and that there were several members of that group. I'm not sure why that mattered, but I could only get it to work this way, and I'm good with that.

My milter config file is at /usr/local/etc/mail/opendkim.conf, and the contents are what the other howto shows:

LogWhy yes
Syslog yes
SyslogSuccess yes
Canonicalization relaxed/simple
Domain unixdude.net
Selector iss
KeyFile /var/db/opendkim/iss.private
Socket inet:8891@localhost
ReportAddress root
SendReports yes

I added the following two lines to /etc/rc.conf:

milteropendkim_enable="YES"
milteropendkim_uid="opendkim"

I added the following three lines to /usr/local/etc/postfix/main.cf:

smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = $smtpd_milters
milter_default_action = accept

I generated the key as shown -- opendkim-genkey -D /var/db/opendkim -d unixdude.net -s iss. The selector (iss in this case) can be any string you want. I went with iss because that is the system name on which this is running. This places the files in /var/db/opendkim, named iss.private and iss.txt.

You create two records in DNS; for my server, I created iss._domainkey.unixdude.net as a TXT record with the contents of iss.txt above:

[root@iss(F12):/usr/local/etc/mail]host -t txt iss._domainkey.unixdude.net
iss._domainkey.unixdude.net descriptive text "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDNKaHQWGIuv7uCByXo9/gQon4BAYKXddNS3oS21tXszTz+Z+BN+ROqZSMKyScdqYdn+dP8TBTnWENKV1BCcJLDRLXi8Nmkiafm4MAswxBtPVRwanJVgHAgPqRuy8KARI/I7LmOt4ZxGkngLgfpqC0BXUBDTIOQJNw+GZJctfKuXQIDAQAB"
[root@iss(F12):/usr/local/etc/mail]

I also created _adsp._domainkey.unixdude.net as a TXT record, with the following contents:

dkim=unknown

When I test the key, I receive a message saying that the key is not secure, and my google searches indicate that this means the server is not using DNSSEC. I'm okay with this.

[root@iss(F12):/usr/local/etc/mail]opendkim-testkey -vvv -d unixdude.net -s iss -k /var/db/opendkim/iss.private
opendkim-testkey: key loaded from /var/db/opendkim/iss.private
opendkim-testkey: checking key 'iss._domainkey.unixdude.net'
opendkim-testkey: key not secure
opendkim-testkey: key OK
[root@iss(F12):/usr/local/etc/mail]

After I checked the key, I started/reloaded the services:

service milter-opendkim start
service postfix reload

After that, I was off to the races, and I can now email my wife's mac.con address again.

FT-7800R install, continued

2022-10-03T00:00:00-04:00

Continuing my earlier post about my Yaesu FT-7800R install, I completed the radio portion this weekend, so now I can get on the air.

I had a Diamond NR770HNMO antenna, and bought a fender bracket mount for my F150. All I needed was an NMO mount and to fish the coax line to the radio.

A local ham friend had an NMO mount and a connector, so on Saturday I went to his house to complete the install.

After some doing, we got the coax through the firewall, then we ran it all the way to the back of the cab, where the radio was mounted. We crimped on a connector, then discovered that the cable was too short. By an inch.

We noticed that we ran the cable in and out of a bracket in the front footwell, so I removed one of the bracket's mounting screws and pushed the cable around the bracket. That gave us just enough cable length to reach the radio without relocating the radio. Crisis averted.

Then we turned it on, and we discovered that its sensitivity wasn't up to par.

In the end, my friend gave me a loaner Kenwood 2m radio so I would at least be able to get on the air.

When I got home, I did some experimenting with another local ham, and I found that yes, the Kenwood is definitely a little bit more sensitive, but the Yaesu seems to be working fine at my house. I'll use the FT-7800R for a couple weeks, then decide whether to keep or replace it. If I replace it, another radio should be very easy to install.

The only necessary remaining piece for this install is to build or buy a mount for the faceplate. I'm thinking of building a wooden something that mounts to the front seat bolts and holds the faceplate in front of the center console.

In addition to that, I still want to hide the controller and speaker cables under the carpet.

Anyway, now on to the pics of what was completed this weekend.

First, the mounted antenna. It's a little tight getting the coax through the crack between the fender and the hood, but it seems to be okay.

The antenna wiring heading toward the firewall:

The wires going through the firewall:

The radio in the back of the cab:

The remote face plate in the front:

The remote face plate and mike in the front:

Update: A friend pointed out that I missed a couple obvious photos here, and I was also asked about how tight the fit is between the hood and the fender.

First, a pic showing the antenna (it is difficult to photo since it's so thin).

Now a couple pictures of the mount, including a closeup:

Finally a profile picture of the truck:

FT-7800R install

2022-09-25T00:00:00-04:00

This is not my typical type of post, but as I have said, communications and networking are things that excite me greatly. It's why I collect antique telephones, why I enjoy scanner and shortwave listening, why I learned the OSI model and have spent much of my career in and around computer networking.

It's also why, over the summer, I studied for, took, and passed the Technician amateur license exam, after which I bought a Baofeng UV-5R. It didn't take me long to realize that the UV-5R does not receive anything at my house, so I quickly replaced it with a Yaesu VX-6R, which I am finding to be an excellent HT.

I also bought a Yaesu FT-7800R to install in my F150. I finally got around to that install today, and I know that there are lots of questions about how to install a ham radio into a vehicle, so I am documenting my install in this post.

First, there is a grommet with a nipple on it on both the driver side and the passenger side of the F150. I started my install by cutting the nipple slightly and running a fish tape through the firewall through that grommet.

As expected, the fish tape came out in the footwell:

I ran the power cable through the grommet to the back of the truck, where I am installing the FT-7800R. Conveniently, the F150 has cable raceways under the door trim:

In this picture of the raceway you can also see where the faceplate and speaker cables leave the trim and go under the seat:

At the B-pillar, I came out of the cable raceway but stayed below the trim. I tried but was unable to push all the way through. (I did not use fish tape for this part.) It's possible that the cable raceway only exists in the door wells, or that the bundle of cables bunches up at the pillar, with some of the cables go up the pillar.

At the C-pillar, I pushed the cables under the trim and around the seat belt bolt:

I used L-brackets to mount a piece of wood to the passenger-side rear seat belt bolts. It works very well, and is solid, even without any nuts holding the board in place. The seat does push slightly against the radio when the seatback is moved up or down, but during normal use there is plenty of room behind the rear seat for the radio.

In addition to power, I ran the speaker and faceplate cables through the cable raceway. Those cables come out right at the B pillar and go under the front seat. I would like to hide those cables under the carpet, so they are only visible under the seat.

The engine compartment has lots of places to reach ground. I chose this center-mounted bolt:

I added an ATO fuse holder to the positive cable right at the battery, and I zip tied the positive power cable to other things along the way to the grommet:

I do not currently have an antenna or a mount for the face plate, but I will be fixing both of those issues this week. For my first test, I used a ground plane antenna I made about a month ago.

I received an excellent signal report on a repeater that I cannot hit with my HT, so I count this as a win.

There are two more things to do. First, I need to install a permanent antenna; the plan is to use an NMO mount on the roof. Second, I need a mount for the faceplate and microphone; the plan for that is to get the FT-7800R version of the Lido LM-300-1001.

I should call out that I forgot to install a ferrite core in the power feed, so I may suffer from engine noise. If I do, I will splice the line and install a ferrite core.

Using the program map type in autofs

2022-09-15T09:50:00-04:00

I recently learned about the program map type in the Linux automounter.

You can read about it here, but there is little information about it, and I was unable to find any web pages about it, so I am writing this post since it might help others.

The NFS automounter on Linux can take a static map, which we all know, including wildcard. The static map is in the format of key options export_location.

Well, the automounter also supports a program map type. The program map type specifics a program to run, and the program returns the export location for any key in that mount location:

/mnt/nfs program:/usr/local/sbin/mapper.py

It's really quite simple: The program takes the key as argument 1, and returns the export_location that would be found in a static map. The program does not return anything else -- it returns only the location of the NFS export. If the program returns anything else, autofs reports an error in the logs. The documentation does say that it can return "everything but the key," but this was not my experience.

In my example, if the user executes cd /mnt/nfs/blah, the automounter executes /usr/local/sbin/mapper.py blah, and the script might return something like filer.example.com:/volume1/maps/abc/123, which would then be mounted at /mnt/nfs/blah as expected.

This can be useful if you want a fully dynamic automount map that is maintained by a group other than the sysadmins of a server. In my case, a development organization maintains a list of exports in a file that the script retrieves with HTTP. The script then parses that list of exports in the program, and any given key returns the correct location for the automounter to mount. The developers can now update their own autofs maps without involving the admins.

Better spam blocking

2022-09-15T09:30:00-04:00

As mentioned previously, I continue to be inundated with spam email. Fortunately, every spam has an observable email format: the email comes in the form of user@word1.word2.TLD.

This allows me to update my blocked_senders file to defer every email in that format with a 450, using this pattern:

/@.*\..*\..*$/ 450 No such address

When I first started doing this, I was manually watching inbound email logs to see what was deferred. I would then update the blocked_senders file to add the domain as a 521 (for obvious spam), or with an "OK" for obviously valid email or unknown spam. Then I rebuild the blocked_senders map and reload postfix.

After a while, I noticed that the domain (word2.TLD) is always a parked domain. I wrote a script to check each domain found in the deferred email logs to detect whether the domain is parked. With this information, the script makes the decision and updates postfix for me: Parked domains are added to blocked_senders with a 521, and other domains are added with an "OK" to allow my server to receive the mail.

This is basically a simple knock-to-enter form of blocking, and so far it is working very well.

So now I get emails like this, as my scripts do their work:

Adding spam domain: [mydealmenia.com]
Adding valid domain: [mail.goodreads.com]
Adding valid domain: [readaloudrevival.com]

All totally automatic now.

The code I used can be found here.

Update

2022-08-05T00:00:00-04:00

I've been busy at home, but not working in my home lab. Lately I have been preoccupied by a few things.

First, I have been busy with prepping -- I can see how things are going, and I predict we will need stores of supplies, so I have been busy building those stores of supplies. Specifically, I have been going through The Prepper's Roadmap with my wife, and we have been learning a lot.

Second, I studied for and passed the technician class amateur radio license exam. I bought a Baofeng UV-5R and find that I am too far from the local repeaters to use it, so I'll sell this and find a better HT. I also picked up a Yaesu FT-7800R for my truck, and hope to get that installed soon.

Third, my family and I traveled to Orlando for a week at Disney World -- we went to the Magic Kingdom and Hollywood Studios, but unfortunately we missed EPCOT since I got sick and have been sick for the last week. We hope to head down to EPCOT sometime in the next few months to make up for the day we missed. Speaking of Disney, Galaxy's Edge is quite possibly the best themed area I have ever visited, and Rise of the Resistance is without question the best theme experience I have ever had in my life.

Back to the update... We have more prepping to do -- skills to learn, materials to gather, and bags and vehicles to pack -- but after that I hope to get back to my home lab.

Dealing with spam

2022-08-01T00:00:00-04:00

Somehow I have been added to several mailing lists, all run by the same organization. The emails look legit, and are ones that I might have subscribed to, but I cannot get off of them no matter what I do -- it seems that they ignore all requests to be removed from the list. Postfix to the rescue: Postfix has a smtpd_sender_restrictions setting, which allows the user to deny specific senders -- but the caveat is that it uses the envelope sender, not the "from" header in the message.

I took note of the envelope senders, and have added those to a blocked_senders file:

/6045-.*-2315@.*redacted.com/ 521 No such address

My smtpd_sender_restrictions configuration includes the line check_sender_access pcre:/usr/local/etc/postfix/blocked_senders, which includes the single line above. Earlier versions of my blocked_senders file had multiple rows for this one sender, but I finally realized that the first and last numbers (6045 & 2315) are the account number, and that the middle number was an ever-increasing campaign ID.

Now, when mail comes in from those addresses, it is rejected, and I don't ever see it. And, for some reason, this particular sender keeps sending me a half-dozen emails a day, despite getting 521 errors for every email.

More than this, though, I have noticed that spammers have gotten much more sophisticated: they register real domains and run their spam servers on static IP addresses with real A & PTR DNS records, so they look legitimate and they pass most of Postfix's spam tests. I mark every such email as spam, and I add the envelope sending domain to the same blocked senders file mentioned above. Every day this blocked_sender config catches multiple attempts from these same spammers. Until they register more real hostnames and domain names, I might be good on spam blocking again!

Learning Python

2022-05-20T00:00:00-04:00

I have been programming in Perl since the early 1990s. My early scripts were what you would expect them to be, but my skill of course increased over time. I have never found a thing I wanted to do that I could not do in Perl and/or shell scripting (Bourne when portability is needed, Bash when it is not).

For my job, I need to learn Python, and a friend recommends the book Automate the Boring Stuff with Python. In addition to the book, the author also has a Udemy course, and I am working my way through that right now. One of the examples given in the course is a Tic-Tac-Toe game.

Earlier this week, as I was going through the course, I had WarGames playing silently on the TV. Right when the movie got to "put X in the center square," my course also got to "put X in the center square." See attached pic ... not faked, and not timed. It just worked out this way.

Anyway, I decided to write a Tic-Tac-Toe game in Python, and seeing as I have 1- never written a game in my life, and 2- never written a line of Python, I think this is a pretty good first attempt. The only thing I took from the course example was the idea for the hash data type; 100% of the rest of this was my own.

NAS shakeup

2022-05-12T00:00:00-04:00

For my storage needs, my first NAS was a Synology DS415+. This was my primary NAS until I purchased a DS1618+, at which time it became my backup unit. When I filled up the DS1618+, I had a few options: buy a small NAS and use some 4TB disks I had on hand, or buy bigger disks, or buy a NAS with more storage bays. I chose to buy a DS220+, into which I shoved two 4TB drives, and this became a second primary onsite unit.

A couple weeks ago, the DS415+ died, leaving me with no local backup. (I have a DS118 offsite at a friend's house, which is used as one of my two offsite backup targets.) I purchased a DS1821+ as a new primary system, and the DS1618+ became my local backup unit. All of my NASes are set up for SHR (primary is SHR2), and all of my NAS transitions have been as seamless as one would expect from Synology: the process was as simple as: move the drives, (re)install the OS, and go.

I will be selling the DS220+ soon, because I have added yet another spare 4TB drive to the DS1821+, and am in the process of relocating data from the 220 to the 1821. This also frees up the two 4TB drives that were in the DS220+, and those will go into the other two NAS units.

I never like losing a NAS, but it's nice that this worked as well as one could hope. As a bonus, I got an upgrade to the latest offering from Synology, which also allows me to free up another drive bay: I can switch from SSD cache to NVMe cache -- when I buy an NVMe drive or two.

ECMP in OSPF

2022-02-20T00:00:00-05:00

I have recently switched most of my offsite network connections to WireGuard, away from ZeroTier. This is great for all of my systems except for one: the NAS (a Synology DS118) I have at a friend's house for offsite backup. That NAS can run ZeroTier and WireGuard, but I don't know of any dynamic routing protocol options for it, so I can have only a single WireGuard connection rather than the two I want. My solution here is to create a ZeroTier network between that NAS and the two VMs I have at DigitalOcean. For my purposes, that is close enough to having two WireGuard links and the ability to route to both of them.

Another issue I wanted to resolve was the CPU load on my USG: with WireGuard running on the USG, high traffic utilization on the WireGurad links results in 100% CPU load on the USG, and that slows everything down. The solution there was to add WireGuard links from my Pi-Hole systems at home to my DigitalOcean VMs, and to increase the OSPF link cost from the USG to the DigitalOcean VMs. As long as my USG is up, I'll have connectivity to my DOVMs, but if one of my Pi-Hole systems is up, I will have better connectivity to the DOVMs.

I also wanted to set all of this up with redundant links and ECMP such that any link can be down, and have everything continue to work.

Below is a current network diagram. The offsite links are WireGuard with the exception of the ZeroTier network (shown in green) that connects the two DOVMs and the NAS. (The two DOVMs are of course also connected by a WireGuard link.) A single OSPF area is used throughout, for all of the routing.

This has been an enjoyable project, and a great learning experience.

OSPF update, Cable cleanup, and finished rack

2022-02-11T00:00:00-05:00

A quick update on my OSPF cutover. With one exception, everything has worked great. The one exception is that OSPF would not work over the WireGuard link between my USG and my primary DigitalOcean VM that runs FreeBSD 12 and FRR7. I thought that was odd, since OSPF worked fine over WireGuard between the USG and another VM at DigitalOcean, running Ubuntu 20.04 LTS and FRR8.

After many web searches, I found this page that mentions the two requirements to getting OSPF working over WireGuard:

The WireGuard interface must be set to Non-Broadcast network type
OSPF neighbors must be statically configured

Once I did that on both the USG and the FreeBSD/FRR7 system, my two routers neighbored correctly, and my FreeBSD/FRR7 VM obtained the entire LSDB.

Two weeks ago I finished racking everything in my new rack, and last weekend I cleaned up the cables behind and below the rack.

As you can see from these "before" pics, the cables were a mess. There was way too much slack, and just a jumble of cables below the rack.

This jumble has been there for at least a year, and I finally bundled the cables and moved the slack to the other side of the wall. I think the result is pretty awesome.

The finished lab looks pretty good. Many thanks to Hayden James for the inspiration here. You can see my progress here.

For those who are interested, here's what's in and around that rack:

U12: TRENDnet 24-port Keystone patch panel, with VCE inline couplers and VCE blanks
U11: Ubiquiti USW-Lite-16-PoE and Ubiquiti USG
U10: Cisco 2611 (Lab R1)
U9: Cisco 2610 (Lab R2)
U8: Cisco 2610 (Lab R3)
U7: Cisco 2960 (Lab S1)
U6: Cisco 2960 (Lab S2)
U5: Cisco 3750 (Lab S3)
U3-U4: Shelf holding Raspberry Pi 4B (8GB) and Lenovo M900 Tiny
U2: Empty
U1: Pyle PDU

I left U2 empty so that I can reach the latches that allow my Networx rack to swing out. Similarly, there is no shelf at U11; the UI devices are simply resting atop the Cisco in U10.

Below the rack are my Synology DS220+, a Grandstream DP750, a Linksys PAP2, and a UPS for the infrastructure (not the lab devices).

The monitor above the rack is connected to my Raspberry Pi 4B. I would like to use it as some sort of dashboard, but I haven't done anything with that yet. That's a project for another day.

Lab update

2022-01-31T00:00:00-05:00

I have been hard at work these last few weeks in my home lab, and have done quite a bit. Here are a few of the things I have done:

Installed sshwifty, a really cool, useful package that a friend pointed me to.
Built a T1 crossover cable to connect the two routers I have in my home lab that have a T1 CSU/DSU WIC installed.
Installed Portainer on my production Docker Swarm
Renumbered my home network from 192.168.1.0/24 to 192.168.8.0/24. I undertook this effort because I have a NAS hosted at a friend's house, and his network also uses the 192.168.1.0/24 address space, so that made it difficult for me to get to my NAS -- sure, I could get to it, but every system that needed access to it needed to be on my ZeroTier storage network. Now, I can route directly to it, and return packets are routed to my network instead of to his.
Deployed WireGuard extensively, using FRR, OSPF, and Anycast to access systems. Because of CPU utilization issues I discovered after doing this, I might move my WireGuard routing/endpoint to my raspberry pi or to a VM in order to work around those issues.
Added more monitoring under Kuma.
Deployed a new NAS, a Synology DS220+ to take some load off my DS1618+ and to use it as cold storage.
Through the generosity of a friend, I have replaced my three Cisco 2950s with two 2960-Xs and one 3750G.
I have finished racking all my devices.
I have cabled up my Cisco lab again. I have also connected the lab to my home network via my Ubiquiti USG's LAN2 port, so that I can route to it. I have console access to all of the devices, using a 4-port USB-to-serial adapter and some mini USB cables (for the 2960s) connected to my Raspberry Pi, but I also want network access. The current lab setup, which will hopefully be useful for much learning, is shown here:

I installed a patch panel, which cleaned up the rack a bit. Here are before & after pics. Because my switch and router are not rack-mount devices, it's not as clean as it could be, but I still think this came out very well.

Next up:

Configure the newly recabled Cisco lab.
Finish migrating data to the DS220+.
Clean up the power and ethernet cables behind and below the rack.

Goodbye BGP, hello OSPF

2022-01-12T00:00:00-05:00

Soon after I learned about Anycast, I employed it on my home network, so that I can have multiple instances of Pi-Hole, mostly so that I can take down my Raspberry Pi without hearing from the family about how the Internet is down.

Until last night, I was using BGP (as part of FRR) for Anycast, running FRR on my Raspberry Pi and some other devices, and peering those with each other and with my router.

Well, I got tired of the full mesh that BGP requires, so last night I switched to OSPF. It still supports Anycast, but does not require a full mesh of routers, making it much more suitable to my needs. I mean, of course I knew BGP was not the right long-term answer, but it was easy and all of my coworkers were able to help me with any configuration questions -- and I got to say that I ran BGP at home, which was fun.

I have been using ZeroTier as my VPN solution ever since a friend mentioned it. I want to have some redundancy in case ZeroTier goes down, so last night I configured WireGuard on my Ubiquiti USG, as well as one of my offsite systems. I prefer to run this on my router anyway, since I don't want my whole network to go down if my Raspberry Pi goes down. If my network goes down because my router is down, well, then I have bigger problems than not being able to access my remote systems.

So, I'll use WireGuard as the primary, and ZeroTier as the backup.

One of my main goals right now is that I want to retire my Rasperry Pi 3B unit in favor of a PoE-powered Pi 4B unit, freeing up a power oulet, reducing cables, and generally simplifying my setup.

I'm not done yet, but I'm making good progress on these goals.

MacBook Pro recovery

2022-01-04T00:00:00-05:00

My work laptop, a 2018 MacBook Pro, needs a new battery and so I will be shipping it off to have that serviced. Prior to getting that serviced, I need a temporary system, so I went to the office to grab one of the spares we have, another 2018 MBP. This system was used by a former coworker, one who retired months ago, and he no longer remembers the firmware password he configured on the system.

I thought this system was a brick, until an Apple expert pointed me to the DFU restore option using Apple Configurator 2. The startup key sequence for this is ... tricky. I tried it a few times, but was not successful until I watched this YouTube video that demonstrates the process.

Using DFU mode, I first tried the Revive option. The laptop booted back to the firmware prompt, so I then tried the Restore option, which caused this laptop to boot into Internet Recovery mode. Perfect! Now I have this unit installing its original OS -- Mojave -- and will update to Big Sur once it is done.

Internet Recovery is cool, but slow. It's a little faster over ethernet versus wifi, especially with a gigabit connection, but it's still slow. Anyway, this thing is working now and is no longer a brick, so it's good now.

Upgrading to DSM7

2022-01-03T00:00:00-05:00

I've been meaning to upgrade my Synology NASes to DSM7. I upgraded my DS415+ last week, and it went smoothly, so today I researched all the packages I have installed on my primary unit, a DS1618+. I found that all but two would upgrade seamlessly; the two trouble packages were Plex Media Server and ZeroTier. Both of these are supported on DSM7 -- see instructions for PMS and ZeroTier -- but are more involved than simply installing a package. In particular, PMS requires a very specific procedure to migrate from DSM6.

I had ZeroTier up and running quickly and easily using the linked instructions, but Plex Media Server gave me a lot of trouble, so I figured I would post about it in case others run into the same issue.

I had PMS running on DSM6, and I wanted to migrate my installation to DSM7. This is not as simple as upgrading a package, so I took care to perform the upgrade as instructed. Following the instructions linked above, I downloaded PMS directly from Plex, then I began a manual installation.

During the installation, the wizard presented an error indicating a failed install (sorry but I did not take a screenshot). I then started doing lots of research and tried the installation several times. I also saw the migration log at /volume1/Plex/Migration.log, and it showed success, so that gave me an idea: maybe the installation really did succeed, even though the wizard showed a failure. I cleaned out everything (did a remove+erase of the PMS package, restored /volume1/Plex, removed /volume1/PlexMediaServer) and reran the installation. Predictably the wizard showed a failure, and I just left it and tailed the migration log until the log showed success:

Plex Media Server migration to DSM 7 started:  Mon Jan  3 15:37:31 EST 2022
=== Mon Jan  3 15:37:31 EST 2022 === Start: Change ownership
=== Mon Jan  3 15:37:38 EST 2022 === Completed: Change ownership
=== Mon Jan  3 15:37:38 EST 2022 === Start: Convert symbolic links
=== Mon Jan  3 15:38:55 EST 2022 === Completed:  Convert symbolic links
=== Mon Jan  3 15:38:55 EST 2022 === Start: Migrate Plex Media Server
=== Mon Jan  3 15:40:42 EST 2022 === Completed:  Migrate Plex Media Server
=== Mon Jan  3 15:40:42 EST 2022 === Completed:  Migrate Plex Media Server
=== Mon Jan  3 15:40:42 EST 2022 === Start: Clean /volume1/Plex
=== Mon Jan  3 15:40:43 EST 2022 === Completed: Clean /volume1/Plex

I exited the wizard, and all was well.

There is one other change I did on the final, successful installation: when I removed the PMS package after the first install (per instructions it requires two installs for a successful migration from DSM6), I also removed the PlexMediaServer shared folder. Maybe that was key: I did not remove that shared folder after the first install, until my final attempt.

Unfortunately DSM7 is complaining about my 3rd party RAM upgrade (2x 8GB sticks that were removed from my Lenovo M900 when I upgraded that to 32GB), but otherwise DSM7 is great, and the upgrade went smoothly with the exception of Plex.

Retiring my Dell R610

2021-12-22T00:00:00-05:00

Ever since a friend gave me a Dell R610, I have used that as my primary home lab server. Due to its age, I find that I am unable to run the most recent versions of some software packages, including the latest iterations of VMware ESXi and vCenter. Also, my unit sits in a small room near the 3rd floor AC return in my house, and this means that the heat it generates is pulled into the AC system, which affects my home's HVAC operation.

For these and other reasons, I have decided to retire the R610 and replace it. A while back, I heard about ServeTheHome's TinyMiniMicro project, so I read some of their articles and decided to purchase a Lenovo M900 with an i7 CPU. I upgraded my unit to 32GB RAM (the max), and this model includes vPro technology, which is a plus or a minus depending on who you ask. This unit will sit in my rack, rather than on the floor leaning against the wall. It will be a huge bonus that the M900 generates less heat and uses less power than the R610.

My new M900 is much less capable overall than my R610: my R610 had a dual 300GB HDs in a mirror, 192GB RAM, and dual Xeon 3.0 GHz for a total of 16 vCPUs. The M900 has a single 256GB SSD, 32GB RAM, and a single 2.8GHz i7 CPU for a total of 8 vCPUs. Even so, I think this was a great purchase based on my use case, which is to run ESXi and a dozen or so VMs. Running vCSA 6, I maxed out on the R610 out at 40GB RAM usage, 10GB of which was vCSA -- so if I avoid vCSA, I'll stay below 30GB RAM usage. Per-thread, the M900 is about 30% faster than the R610, so overall I think this is a huge win.

Having less RAM means I will have to be more selective about the VMs I run. Fortunately, since this type of setup is so cheap, if I need more capacity, I'll just buy another one of these 1L systems, be it a Lenovo or a Dell or an HP. So far, the M900 has exceeded my expectations.

Getting ESXi instsalled was an interesting experience: I couldn't get the M900 to boot from USB or PXE -- it seemed stuck on booting the included Windows installation. I quickly learned the reason for this: the M900 was configured for UEFI booting only; BIOS booting was disabled. I enabled BIOS booting, downloaded the Lenovo custom build of ESXi 7.0u2 and added it to my BIOS-only PXE setup, and started my install. I was off to the races, and so far I haven't found any issues with this build on this hardware.

After I got ESXi 7 installed on it, I installed vCSA 7, then added both the R610 and M900 to that vCenter Server so I could use vMotion to migrate all of the VMs. I had a few VMs on the local storage on the R610 -- notably my UniFi Controller so that I can upgrade my Ubiquiti devices -- and it's so cool that vCenter Server can migrate from local storage to local storage on different servers, without an intermediate stop on a shared datastore. Obviously 256-300GB space is not enough for many VMs; most of my VMs are stored on NFS.

Another thing that really impressed me last night is that all of my VMs migrated without issue: No complaints about CPU mismatch. All VMs are now migrated, but I powered down a few of them to save RAM. With vCSA powered down, this system will be perfect for my needs:

I have a lot of cleanup to do here, but it's amazing how quiet and cool this area is, now that the R610 is powered down.

The powered-down R610:

The eBay seller did me right: this thing was super clean and looks practically new:

One last thing: I have decided to convert my PXE setup to UEFI-only: at this point, I have zero need for BIOS PXE booting, and UEFI will be better. That will be an upcoming project.

Wall-mount rack

2021-11-27T00:00:00-05:00

Since I bought my Cisco equipment, I have had it racked in the cheap, two-post desktop rack that came with the equipment when I bought it. It was pretty bad, but it worked:

I really wanted something better than this, so I started doing some web searches to get ideas on what I could do better. Along the way, I discovered Hayden James's home lab beginner's guide, and decided I wanted a wall-mount rack. I did a lot of research about wall-mount racks, and I settled on the Networks adjustable depth, swing-out 12U rack.

The rack arrived last week, and I was able to mount it earlier this week. I still have lots of work to do here, and this is going to be a huge improvement over the original setup.

healthchecks.io

2021-11-15T00:00:00-05:00

Recently, a friend introduced me to healthchecks.io, a service for monitoring cron jobs.
A week or so ago, I configured several of my cron jobs to be monitored through this system, and it has already helped me: yesterday I received a notification that an rsync job did not run, and the reason for that failure turned out to be a filesystem that was unintentionally unmounted. I remounted the filesystem, and today I received a notification that the rsync job finished as expected. What a great service.

Printer update

2021-11-09T00:00:00-05:00

Here's a months-overdue update on that HP Envy printer: I wasn't able to return the printer, and I still wanted a better scanner app than the HP's built-in one, so I decided to take the Raspberry Pi 4 that I mentioned in an earlier post and attach the printer directly to it. The goal of doing this was having better printing than what I was doing before, where the printer was on my IoT VLAN, and my print clients are all on my home VLAN.

In addition to having better printing functionality, I also wanted a better scanner app as I just mentioned. And, I wanted all of it working through Traefik, so that I can access the scanservjs app, printer admin page, and CUPS admin page all through Traefik.

As I mentioned, I got to learn about docker build, because there was no scanservjs image for ARM. Fortunately, building for ARM is super fast on my M1 MacBook Air. I mean, my Intel iMac could build the ARM image too, but it takes a lot less time on my M1 MBA -- a minute or two on the M1 vs 1,000 seconds on the Intel, so I won't be building the ARM image on my iMac again!

I have now automated the build process mentioned in the earlier post: every now and again, I will pull the latest code from sbs20's scanservjs repo, then build it using his Dockerfile, with this script:

#!/bin/bash

IMAGE="ataridude/scanservjs"

[[ -z $1 ]] && echo "Usage: $0 tag" && exit -1
docker image build -t "${IMAGE}:$1" -t "${IMAGE}:latest" . && docker image push ${IMAGE}:$1 && docker image push ${IMAGE}:latest

I know there are ways to fully automate that docker build process, and I'll get to that point at some time, but for now it's nice to have an easy way to update the image.

As you might expect, my Raspberry Pi print server is installed via an Ansible playbook. I should look into the playbook and docker-container-scanner role, since this was the first Docker-oriented role I wrote, and I'm sure there is much that could be improved in the role and the playbook.

Update on migration to Docker/Traefik

2021-11-05T00:00:00-04:00

I have been working hard these last many weeks in order to move all services off of my main DigitalOcean Droplet/VM. In the process, I have been moving everything to Docker, with full automation via Ansible. I have finally achieved this goal, for the web services anyway, and now I can deploy all of my web services with one or two Ansible commands: a bootstrap playbook (needed only for DigitalOcean VMs), and a deploy playbook (good for VMs at home as well as ones at DigitalOcean).

I have uploaded my code to GitHub. For DigitalOcean VMs, I start with playbook_bootstrap.yml, and for all VMs, I run playbook_blog_server.yml.

This single command now sets up everything that used to be an Apache vhost on my main VM -- and all "vhosts" now run through Traefik.

This is a good milestone to reach, because it means I have now accomplished my goal of treating these services as "infrastructure as code": I no longer make changes on the web server -- all changes are done in Ansible and/or Docker, then pushed out to the server.

Next up, I will work on moving my mail server (postfix, dovecot, spamassassin).

While working on this project, one of the things I ran into is that the Pelican search plugin I have been using, tipue_search, is no longer supported. At first, this kept my main site on Apache, but eventually I decided I needed to get this sorted. I made my own copy of tipue_search and added it to the Dockerfile I am using to build this site, and now I have it working in my Dockerized blog implementation. I had considered switching to the brand-new pelican-algolia plugin, and while I still think that is a great solution I decided to stick with the tipue_search plugin I've been using for years.

This Dockerfile uses a multi-stage build process, which I like because it lowers the requirements for my build server: the multi-stage build means that Pelican is not actually installed on my build server -- the entire build is done in a container. I got this idea from a friend, who does this with his blog.

Hopefully I will now find it easier to update this and my Atari blog.

MinFS

2021-10-28T00:00:00-04:00

Over the last month, I have been continuing my work at automating my systems and have made a lot of progress in that effort.

Part of that will be relocating some files that I still want web-accessible, but not on this website. Enter DigitalOcean Spaces, and MinFS to access it.

Set was super simple: After installation, I edited /etc/minfs/config.json to set my Spaces access key and secret key, then added the /etc/fstab entry, created the directory, and mounted it.

/etc/fstab:

https://nyc3.digitaloceanspaces.com/unixdude /mnt/unixdude minfs defaults,cache=/tmp/unixdude-minfs 0 0

[root@ansible-test(U20):67:/mnt/unixdude/ataridude]df -h .
Filesystem      Size  Used Avail Use% Mounted on
MinFS            64T     0   64T   0% /mnt/unixdude
[root@ansible-test(U20):68:/mnt/unixdude/ataridude]ls -l
total 0
-rw-rw---- 0 root root 67748394 Oct 27 22:57 atari-5200-jumpy-video.mov
[root@ansible-test(U20):69:/mnt/unixdude/ataridude]

The URL for that file is: https://unixdude.nyc3.digitaloceanspaces.com/ataridude/atari-5200-jumpy-video.mov

Pretty slick. I'll be adding more to that space over time.

Also: 64T - wow!

acme.sh and Let's Encrypt certificates

2021-10-18T00:00:00-04:00

I have been needing to cut over to acme v2 for a long time, and I didn't bother doing it until way too late -- this weekend. As a result of my tardiness, my iPhone has not been able to receive email from my mail server for more than a month now.

My current mail server runs dovecot + postfix (plus some other stuff), manually installed on FreeBSD, and I have been trying to get Docker Mailserver working on a new Ubuntu server and have been having nothing but trouble with that, so I finally gave up. On the new setup I will still install everything with Ansible, it just won't be Dockerized, and I'm okay with this.

Tonight I decided that in the interim I needed to at least fix the SSL certificates. While doing this, I ran into a problem that I cannot believe is not called out anywhere, so I'm writing about it. To be fair, this is documented here, but no howto I found mentions it.

Let's Encrypt's client page lists acme.sh, but does not bother to mention that one must pass in the --server parameter in order to use the Let's Encrypt CA with acme.sh. One must do this because the default CA for acme.sh is ZeroSSL.

In fact, none of the dozen or so howtos I read made any mention of this! And, since I had never heard of ZeroSSL until tonight, I had no idea that it was a competitor to Let's Encrypt.

In order to create the Let's Encrypt certificate for my mail server (mail.unixdude.net), I set the DO_API_KEY environment variable, then ran:

acme.sh --issue -d mail.unixdude.net --dns dns_dgon --server letsencrypt

The magic there, for the Let's Encrypt user, is the --server letsencrypt parameter -- because as I mentioned the default is ZeroSSL. If you try it without specifying the server, parts of this work and other parts of this do not work, and the problem is not obvious, or at least it was not obvious to me.

In any case, I now have new certs running, and have a fully functioning system again -- which gives me time to cut over to a new server.

I hope this post helps others who are running into the same issue I was hitting.

Blog automation

2021-09-26T00:00:00-04:00

This is a short post after a lot of work.

I have been busy these last few weeks automating the installation of Docker, Traefik, and my Atari blog with Ansible. I have reached the point that I can take a fresh install of Ubuntu 20.04 LTS and end up with a completely functional blog server. The "fresh install" requires only my user, SSH key, and passwordless sudo configuration... once I have that, Ansible takes care of everything else. I simply run ansible-playbook playbook_blog_server.yml ... et voila.

Now to work on correcting some of the things I did wrong here, and moving the unixdude.net blog into a Docker container behind Traefik.

For those who are interested, I have pushed the code.

Docker build

2021-09-07T00:00:00-04:00

I have been having a rough couple months with printers: My Samsung C410w is getting old, so I wanted to replace it. My wife recently heard about HP Instant Ink, so she wanted to try that service, but I decided to buy a monochrome Brother printer -- which was great except my wife decided she needed to be able to print on cardstock, which the Brother could not do. So, we returned the Brother and picked up an HP Envy 6055 -- and proceeded to print more than 750 pages last month. Suddenly, Instant Ink doesn't look so affordable.

Anyway, in order to go back to a laser/LED printer, I need to find a replacement to the Envy's scanner. Well, for 15+ years I have had a CanoScan LiDE 30, which still works fine. I had been using it for years with Mattias Ellert's excellent TWAIN SANE Mac utilities, but Mattias has not maintained this for 4 years now, so I switched to using scanimage on a Raspberry Pi.

Scanimage is fine for me, but there's no WAF there -- and now that I need to increase the WAF of my scanner setup, and now that I'm learning about Docker, I wanted to create a Docker image containing a scanner utility that I could run on a Raspberry Pi connected to the scanner, which could then be left in a convenient location for my wife to use.

In doing some research for this project, tonight I found this nice front end, which is also available as a Docker image. Unfortunately, that Docker image is only available for amd64 architecture -- but of course my Raspberry Pi is arm64.

I next tried building the image on the raspi, using the Dockerfile in the github source, but even with an 8GB 4b model, 25+ minutes and it still had a ways to go -- and, it ran out of disk space.

Thankfully, I happen to have a M1 MacBook Air, so I fired up the new M1-specific version of Docker Desktop on that, and built an arm64 image -- in about 30 seconds!

I quickly pushed that to Docker Hub, then easily ran it on my Raspberry Pi with:

docker container run --device=/dev/bus/usb/001/003 --name scanner -p 8080:8080 --rm ataridude/scanservjs

I'm not done yet, but this is a good place to stop tonight.

Traefik

2021-08-12T00:00:00-04:00

Lately I have been learning about Docker, and I want to convert my current setup to Docker, where it makes sense to do that. I currently run about 20 vhosts on this Digital Ocean droplet, and at least one of those has a MySQL back end. I also run Postfix/Dovecot/Sieve for mail. And, some other things run on here.

As part of my Docker studies, I was talking to a friend recently about my setup, and we were discussing whether Docker is the right solution for the things I mentioned. For now I'm going to hold off on using Docker for my mail server, but I know there are some container-based mail server solutions; I will look at those later, once I have migrated at least some of the vhosts.

My friend and I started by discussing the vhosts I run. Enter Traefik, a free, Docker- based and Docker-oriented proxy solution. Even better: containers are automatically registered with Traefik, and from what I have seen so far the only configuration necessary is to set some tags on the containers as they are run, in your docker-compose file.

I decided to take one of my vhosts -- www.ataridude.net -- and run it in/with Traefik. I had some good success last night, and while it is not completely ready yet, you can add this host entry to your system if you want to see it before it goes live:

159.65.169.236 www.ataridude.net

To get this up and running, I built a docker-compose file based off of the one found in the Traefik quick-start:

version: '3'

services:
  reverse-proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.4
    # Enables the web UI and tells Traefik to listen to docker
    command: --api.insecure=true --providers.docker
    ports:
      # The HTTP port
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock

  atariblog:
    image: ataridude/private:atariblog
    labels:
      - "traefik.http.routers.atariblog.rule=Host(`www.ataridude.net`)"

The observant reader will note that the Traefik admin console is open and available. Ordinarily this would not be a good idea, but port 8080 is only available via ZeroTier (my Digital Ocean firewall configuration blocks port 8080), and if someone joins my ZeroTier network, I have bigger issues than the Traefik admin console.

My blogs are Pelican-based, but I'm trying to get away from installing software on my systems, at least as much as possible, so I decided to build the ataridude.net image using Docker multi- stage image builds. This means I do not need Pelican or anything else installed on my server -- I only need Docker.

Here is the Dockerfile I'm using to build that image:

FROM python:3 AS builder
WORKDIR /usr/src/app

COPY requirements.txt ./requirements.txt
RUN pip install -r requirements.txt
ADD content ./content
ADD theme ./theme
COPY pelicanconf.py ./
RUN pelican /usr/src/app/content/ -s pelicanconf.py

FROM nginx:latest

COPY --from=0 /usr/src/app/output /usr/share/nginx/html

I based this on my friend Brad's Dockerfile.

It's not as automated as I'd like, but it does work, and I will continue to improve it.

In addition to greater automation, I plan to start moving vhosts from my FreeBSD droplet over to a new Docker- and Ubuntu-based droplet.

June 2021 update

2021-06-14T00:00:00-04:00

I've been busy over the last few weeks, so there will be several updates in here.

Over the weekend I updated my Ubiquiti switches, which required that I shut down my ESXi servers. The R610 came back just fine, but the raspi4 ARM one did not. The problem on the Raspberry Pi was the known UEFI config corruption issue, when the UEFI config is saved on the SD chip.

I thought maybe I would have to buy a new SD chip, but it turns out that only the RPI_EFI.fd file was corrupt. Unfortunately my previous blog posts did not serve their purpose -- I was not able to use them to reinstall the Raspberry Pi ESXi ARM fling, and had to do all the research again, so:

ESXi on ARM installation howto (there are many; this is the one I used last night)
ESXi on iSCSI on ARM

Since my SD chip was fine -- "diff -r" showed that only RPI_EFI.fd had changed since install -- I copied the RPI_EFI.fd file to the SD chip, then booted the Raspberry pi.

I updated the EFI config as shown in step 3 of the install link above (the step which removes the 3GB RAM limit), then I reconfigured the EFI to boot from iSCSI, as shown in the second link. That post does not mention an issue specific to Synology NASes, so I'll mention it here: when using a Synology NAS, the Boot LUN must be 1, not the default of 0.

After that, I was able to boot my ESXi-on-ARM raspi again. It is currently sitting on my desk, and I'll probably leave it there, since I do not have remote console to it: with it on my desk, I can easily connect a monitor and keyboard when needed.

I decided I wanted to refresh my VMware skills, so I signed up for VMUG Advantage again about a week ago. One night last week I set up vCSA and then a cluster of 3 ESXi VMs. VMware's unsupported-but-works-fine-in-a-lab nested virtualization functionality is awesome, and I was feeling generous so I gave those 3 ESXi VMs 32GB and 8 vCPUs. Since the R610 is not supported on v7 VMware products (the CPU is too old), I'm only running ESXi and vCSA v6.7, but that's good enough for now. It might be time to upgrade soon.

I put those nested ESXi VMs on my lab VLAN, and I have discovered an interesting issue: I cannot use Netboot.xyz with VMs in that cluster. I'm still trying to figure out why, but they just don't work. The ESXi VMs are on VLAN 7 (lab VLAN, 192.168.7.0/24), and any VM on that VLAN that is hosted directly on the R610 will PXE boot to netboot.xyz without issue, and I can install any OS. However, nested VMs (those running on the ESXi VMs) will PXE boot to my menu, and can PXE boot local things, but they will not PXE boot netboot.xyz -- the netboot.xyz menu loads, but no OS can be installed; see screenshot below. Again, if I do exactly the same thing with a VM on the R610, it works perfectly. Bizarre.

One specific reason for signing up for VMUG Advantuage again was that I want to play with NSX again, and to play with vSAN for the first time. Those are both upcoming projects.

Another fun thing I did over the weekend was to telnet to an Ubuntu VM from my Atari 800. There is a really cool peripheral currently available for Atari 8-bit computers called the FujiNet, and using a terminal program, one can use telnet. I have written up more on my Atari blog.

ZeroTier Is Fun

2021-05-18T00:00:00-04:00

I recently installed a Synology DS118 at a friend's house for remote backup using Synology's Hyper Backup solution. The DS118 is on my ZeroTier "storage" network, a network shared by the main Synology at my house -- that way, my main NAS backs up to a local system (that is, something on the same layer 2 network), which makes that part easy. I was able to prime the backup at my house and ensure that everything worked, then I gave the DS118 to my friend to take to his house -- and the backup worked exactly as it did when both units were at my house. This is why I like ZeroTier.

Today I realized that the DS118 is not sending notification emails, and I definitely want it to be able to send email when it has something to report. I enabled notifications on the DS118 and configured it to send me mail. It seems my friend's ISP is blocking outbound email, because every test message failed -- that is, until I reconfigured the DS118 to relay through the raspberry pi at my house, which is on the same ZeroTier "storage" network.

The raspberry pi is my home mail relay, through which all other systems at my house send mail to my DigitalOcean droplet: the raspi and the droplet are on the same ZeroTier network.

So, now I have notifications from my DS118, relayed through the raspi at my house, using two ZeroTier networks.

ZeroTier is fun.

PXE booting on ARM64

2021-04-23T00:00:00-04:00

As previously mentioned, my next goal was going to be PXE booting VMs running on the ESXi ARM Fling. Using this page as a guide, I now have this working.

That guide is pretty good, but I had to do adapt it to my setup.

As instructed, I started by downloading the grub EFI binary and installing that on my TFTP server, at tftp.unixdude.net/grubnetaa64.efi.signed.

I updated my USG to add a new IP subnet and VLAN for the ARM64 VMs, so that I could set up a PXE configuration specifically for those VMs, without affecting the PXE setup I have for x64 systems. Yes, if I ran my own DHCP server, I could have added configuration to it to detect the client architecture and to have it send the correct boot code based on that. This way was easy, and it has the benefit of not affecting my home network in the case that the DHCP server VM is down. (High WAF in the network not going down.)

On the new subnet, I specify the DNS server (my anycast IP address), and I point the "DHCP network boot" options to my TFTP server, with a boot filename of /grubnetaa64.efi.signed. I also specify the DHCP TFTP Server IP address.

On the filer, I extracted the grub.cfg, initrd, and vmlinuz files as instructed. Initially I tried putting those files into a /arm64 directory on my filer, but that did not work for me, so I moved them to the root directory and was able to PXE boot the VM right away.

I added the grub.cfg entry as indicated, modifying it to point to my local copy of the Ubuntu image.

After that, I booted the VM. The first time I booted it, I ran out of memory because I had only given the VM 1GB of RAM. I increased that to 4GB (half the memory of the raspi) and was able to successfully PXE boot a VM and install Ubuntu.

I consider this a success despite its impracticality: I cannot realistically PXE boot VMs if I have to dedicate half the server's memory in order to do it.

ESXi on ARM Fling - Update

2021-04-20T00:00:00-04:00

Not long after my last post, the PoE HAT for my Raspberry Pi died -- to be more accurate, one of the components on the board disintegrated. I don't know if this was a manufacturing defect or the result of a bad solder job related to the fan, or something else. In any case, disposed of that HAT and ordered the GeeekPi PoE HAT. My ESXi-on-ARM setup has been up and running since, connected by a single cable.

I have this system booting from iSCSI, so there are no local filesystems -- the SD card is used only to initiate the boot process, the system boots from iSCSI, and datastores are on the filer and are accessed via NFS. I love the elegance of this setup.

So far, I have created several VMs: Alpine, CentOS 8, Ubuntu, Raspberry Pi OS, and FreeBSD. I haven't really done much with any of this, but in my time with it so far I am impressed with its speed. Speed-wise, it obviously doesn't compare to my R610, but for about $100, it's great for what it is, and it makes an excellent geek toy. And yes, I bought it mostly as a toy, but I do plan to use it for some real-world services: In addition to general OS play, I plan to at least add a VM here to my anycast DNS setup.

I might also build a VM on this server in order to relocate my UniFi controller. I'm undecided because I'm not sure how much I like the problem that involves: if the raspi dies, so does my ability to run my controller. Solving this problem is important since I currently run my UniFi controller in a VM that is hosted on my R610 and is NFS mounted. I do need to find another solution to the UniFi controller issue, since there are two switches between my R610 and my filer -- meaning that whenever I upgrade one of those switches, I have to reloate the controller VM to my iMac. This is not how you should do it!

I haven't installed vSphere in a while, but now that I have two ESXi servers, I might do that again, so that I can manage this entire setup more easily.

ESXi on ARM Fling

2021-04-07T00:00:00-04:00

I recently decided that I wanted to play with the ESXi-on-ARM Fling, so I purchased a Raspberry Pi 4B (8GB). I received my unit so I set about installing ESXi.

I followed VMware's instructions in their "Fling-on-Raspberry-Pi" document. It's not a difficult process, but there are a few things to note about the process found in that document.

One of the first steps is to update the Raspberry Pi's EEPROM; VMware's document says to use the Raspberry Pi Imager to flash an SD chip with the "Raspberry Pi EEPROM boot recovery" utility, but I found this to not be available in the Imager utility. With a little help from my favorite search engine, I found the official documentation on updating the boot EEPROM, which includes commands to do it from a running instance of Raspberry Pi OS. With that I was able to confirm that my raspi is up-to-date.

One thing to note is that the official PoE HAT is not supported (at least, not out of the box), because its fan is controlled by the I2C, which is nonfunctional when running ESXi. Unfortunately for me, I did not know about this limitation before buying the official PoE HAT; I chose to solve this problem by soldering leads from pins 4 & 6 on the GPIO directly to the fan -- I want the fan to always run anyway, so that is a fine solution for me.

The main use for a micro SD chip is to configure the UEFI, so don't buy a large one. In my experience, the micro SD chip is also your main boot device (without one inserted, my unit won't boot). Also, you need a USB drive on which to install ESXi. NFS datastores are supported, so I am mounting those from my NAS. I use one datastore for installation ISOs, and one for VMs.

I am looking into installing ESXi on an iSCSI LUN but so far I have not been able to get that to work with my Synology: the Raspberry Pi connects (I see this on the Synology), but the LUN does not show up in the device list on the raspi.

Be sure to follow the instructions on configuring the NTP client, since it works differently than an x86_64 ESXi server does.

So far I have installed Ubuntu 20 and CentOS 8. This is looking like a great little toy!

M1 MacBook Air

2021-04-06T00:00:00-04:00

I recently decided to upgrade my personal laptop. Until recently, I was using a 2015 MacBook Pro that I bought in 2016 after Apple announced they were bringing the butterfly keyboard to the MBP. My goal for that purchase was for it to last until Apple released a better keyboard -- and it worked. Last month I traded my 2015 MBP for a new 2020 M1-powered MacBook Air, and I could not be happier with this upgrade: The M1 is an amazing CPU, the keyboard is awesome, and the M1 MBA is a phenomenal package.

Because of my love of operating systems (specifically Unix-based ones, obviously), one of the first things I did with this system was to download the technical preview of Parallels Desktop for M1, so that I could run VMs on this new laptop.

So far, I have installed Alpine, Debian, Fedora, Photon OS, Ubuntu, Windows 10, and FreeBSD. Most of those distros/OSes install exactly as you would hope, but a few required some special tweaks.

For Alpine and Photon OS, I found that the Parallels VM must be configured as a Debian VM.

For FreeBSD, I used "Other Linux" as the OS type, and the most important part for FreeBSD is to configure the VM to use only 1 vCPU -- it will not boot with more than 1 vCPU.

So far I am extremely impressed with this M1 unit. With the ability to run such a variety of operating systems in VMs, it will prove even more useful to me.

Lab, and network upgrade

2021-03-12T00:00:00-05:00

I like networking -- always have. However, today, my networking knowledge is only okay -- but I want it to be great! My goal is to achieve CCNP certification in 6 months. To help with this effort, I have added a Cisco lab to my home network. It is old equipment (2x 2610, 1x 2611, 3x 2950) running IOS 11 and 12, but that's okay, and it will soon be upgraded: as I cut over to Ubiquiti equipment for my home network, the 2960s I am using today will move to the lab, and I will add a layer 3 switch as well. The lab routers will also be upgraded to something running IOS 15, but I'm not sure what yet.

I like network diagrams, so for fun, here is a diagram of my current home network:

And here is my Cisco lab:

As you can probably imagine, I have been fun expanding my use of VLANs and BGP in my home network.

I have further plans as well:

Introduce a MySQL cluster. This will use an anycast address (see earlier post on anycast) so that clients can specify the anycast address and not care which server they talk to.
Introduce a router between my lab and my home network. Current plan is to use Cumulus VX for this, because I want to learn Cumulus Linux as well.
I want the lab completely separated from my home network; as you can see from the diagrams above, the lab is connected to my home network. This will change as I introduce Cumulus VX to the mix.

Planned network upgrades:

The 2960 in the middle will be replaced with a Ubiquiti USW-Lite-16-PoE
The MikroTik hap AC in the entertainment center will be replaced with a USW-Flex-Mini and a U6-LR-US
The HP ProCurve at my desk will be replaced with a US-8

At that point, all of the switches and APs will be PoE, meaning I will have only a single place for battery backup of the entire network, and fewer wall warts.

FRR and Anycast

2021-03-11T00:00:00-05:00

FRR is one of the things I have wanted to configure on my Raspberry Pi, specifically to enable anycast for Pi-Hole. The reason to do this is that I do not want my Raspberry Pi to be a single point of failure on the network, especially since it has been connected via wifi, to date anyway. Once I finish my transition to UniFi, it will be moved to ethernet.

Anycast is useful for services like DNS, DHCP, and MySQL, and is easier, better in some cases, than using a load balancer. I wanted to use anycast on my home network so that I can upgrade or reboot my Raspberry Pi, while ensuring that such maintenance activity does not negatively affect users on my network.

I started out by creating a DNS entry:

pihole-anycast.unixdude.net has address 192.168.2.1

This entry is not strictly required, it's just a convenience.

I then installed FRR on multiple systems -- my Raspberry Pi and a VM running Ubuntu 20.04.2 LTS, running on my ESXi server.

On each system, I added a loopback address; on the Ubuntu VM I added the following block to /etc/netplan/00-installer-config.yml:

  network:
    ethernets:
      lo:
        renderer: networkd
        match:
          name: lo
        addresses:
          - 192.168.2.1/32

On the Raspberry Pi, running Raspbian (stretch), I created a file with the following contents, saved as /etc/network/interfaces.d/lo.

auto lo
iface lo inet loopback

auto lo:0
iface lo:0 inet static
    address 192.168.2.1
    netmask 255.255.255.255
    alias pihole

To ensure that this file is loaded, you might need to confirm that /etc/network/interfaces includes the following line:

source-directory /etc/network/interfaces.d

There are two important things to note in the examples above:

that the address is a /32,
and that it is added to the loopback interface.

Next, configure FRR to publish that loopback address to the network. I do this using BGP, and the relevant part of my configuration is:

router bgp 65007
 neighbor 192.168.1.254 remote-as 65007
 !
 address-family ipv4 unicast
  redistribute connected route-map anycast
 exit-address-family
!
access-list 7 seq 10 permit 192.168.2.1/32
!
route-map anycast permit 1
 match ip address 7
!

As you can see, I am using BGP ASN 65007 on my home network. My router's IP address is 192.168.1.254, so I have FRR peer with that address. Note the route map and access list configured to publish the loopback's /32 address.

We can take a look at my Ubiquiti USG's BGP routes for that address, and we see the two routes:

admin@ubnt:~$ show ip bgp 192.168.2.1/32
BGP routing table entry for 192.168.2.1/32
Paths: (2 available, best #1, table Default-IP-Routing-Table)
  Not advertised to any peer
  Local
    192.168.2.249 (metric 1) from 192.168.2.249 (192.168.2.1)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      Last update: Sun Mar  7 23:03:51 2021

  Local
    192.168.2.250 (metric 1) from 192.168.2.250 (192.168.2.1)
      Origin incomplete, metric 0, localpref 100, valid, internal
      Last update: Sun Mar  7 13:29:37 2021

admin@ubnt:~$

And here is its routing table:

admin@ubnt:~$ show ip route     
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S>* 0.0.0.0/0 [1/0] via 172.16.0.254, eth0
C>* 192.168.2.0/24 is directly connected, eth1.2
B>* 192.168.2.1/32 [200/0] via 192.168.2.249, eth1.2, 1d17h41m
admin@ubnt:~$

(some data removed)

Now, anything on my home network can reach 192.168.2.1/32, and it does not matter which Pi-Hole answers the request, because both are configured identically. (I have to manually keep their configurations in sync today, but I am working on that.)

As I mentioned earlier, a setup like this is useful for DNS, as I am doing here. It is also useful for DHCP, where multiple DHCP servers use the same back end, and are all configured with the same DHCP server identifier. Using this configuration, the DHCP service stays up as long as at least one DHCP server is available; the backend database ensures consistency between the pools advertised by the servers.

Speaking of databases, this solution works similarly with MySQL: one can build a Percona cluster, where all servers share a single IP address via anycast. Clients connect to the anycast address, and the cluster keeps all servers updated to the latest changes.

General update

2021-03-10T00:00:00-05:00

It has been a while since I did much of interest in the home lab, but that is changing this week.

I am starting to rework lots of things:

This blog content is now tracked in a bitbucket repo
I am upgrading my home network to UniFi hardware
I am moving to an IaC setup

The first part was easy: create a repo, then push to it:

  git remote add origin git@bitbucket.org:unixdude/blog.git
  git push -u origin master

I should have done that a long time ago, but at least I had it tracked in a repo. Now it is tracked in a remote repo. I will eventually migrate this server to Linux so that I can use containers; right now it runs FreeBSD.

The second part comes after much consideration of what I wanted to do. My 3-story house has many WiFi dead spots, and I'm tired of that: it means that my Airport Express and Ring doorbell often don't work well. I decided to go with a UniFi setup, mostly for the single management pane and the ease of adding and upgrading hardware. I should have the new hardware sometime this week, and I hope to get it installed soon after I receive it.

Regarding the IaC setup, that is something several friends have done, and is something I should have done sooner.

I'm working toward having more cattle and fewer pets, and I'm working toward automated installs of that cattle. I'm starting with Pi-Hole, which I use on a pet Raspberry Pi today. When I'm done, Pi-Hole will be installed on the raspi as well as a VM, and those two systems will share an IP address via anycast, so either one can go down without affecting users on my network.

I'm also working toward using containers for things like Pi-Hole. All of this will be installed and configured via Ansible.

Speaking of Ansible, it is taking over some of the responsibilities of my PXE setup: my PXE setup now defaults to a very basic install of Ubuntu 20.04.2 LTS, and Ansible will now be responsible for installing and configuring packages such as NTP and other user- and system-configuration.

NFS automounter

2019-09-16T00:00:00-04:00

In an earlier post, I mentioned the NFS automounter. I make use of the NFS automounter everywhere. And by "everywhere," I mean "nearly every Unix system I install." FreeBSD, NeXT systems, Mac OS X, Linux -- they all get the configuration.

The NFS automounter is useful because of the ease of use: if I need a file, I simply cd to the correct location and use the file.

I install my automounts in /auto, so I cd to /auto/documents or /auto/downloads or whatever. My ESXi server uses files that on other systems are located under /auto/esx.

To make best use of this, I use the same user ID on all systems, and for security best practices, I set root squash even though I'm the only user on the filer.

Also, because my filer is a Synology, I am able to share audio/video between iTunes and Synology's DS Audio/DS Video. I do this by storing my iTunes library under /auto/media, and I set DS Audio and DS Video to use files found under /volume1/media on the Synology.

The NFS filer is also useful remotely, because of my use of ZeroTier. The NFS automounter, with its automatic disconnects, works particularly well in this type of situation.

Home Network update 2019-09

2019-09-10T00:00:00-04:00

My home network has changed significantly in recent months.

First, I purchased three MikroTik devices, an RB260GS for my office and two hAP AC, one for my office and one for my garage. This allowed me to simplify my network and it provides for several changes. My network now looks like this:

As a result, I can upgrade the Synology RT1900ac without taking down my entire home network. I can upgrade the office switch or hAP AC without affecting my family's network access, and updates to the garage hAP AC will only affect my R610-based home lab.

A while ago I was introduced to Zerotier. I gave it a look and found an extremely useful networking solution. To use it, one need only do a few simple things:

Create a Zerotier login
Create a network
Install the software on your systems/devices
Join those systems/devices to your network
Approve the join requests (if your network is private)

The Zerotier admin UI shows the IP addresses of each system, and it allows the user to set a nickname for each device.

Using this solution, I was able to completely disable global SSH access to all of my systems: now, the only way to SSH to my systems is to be on my Zerotier network or my home network.

I created a services VM that sits on both ZeroTier and my home network; this VM forwards packets between ZeroTier and my home network, so with a few simple routes added to a few devices, I can now access my entire home network from anywhere:

Synology RT1900ac has routes to the Zerotier subnet (via the services VM) and the lab subnet (via the hAP AC in the garage)
My laptop has routes to the home and lab subnets, via the services VM

macOS has long had a very cool feature where you can use per-domain DNS servers, so I use my home-based DNS servers to resolve *.unixdude.net no matter where I am. This, coupled with these new routes, allows me to resolve everything at home by hostname, and to connect to it. This is immensely useful.

SSH, screen, and tunnels - oh my

2019-08-28T00:00:00-04:00

I was recently asked about SSH, tunnels, and reconnecting disconnected sessions. Some people use mosh, but I take a different approach, one that does not require any special server software on the target server.

As with mosh, I recognize that sessions will become disconnected. As such, I run a screen session on a server in the data center, and I initiate all SSH connections from that screen session. I use screen instead of tmux, mostly because it is extremely simple to create a new screen tab/window from the command line. I have created aliases to run a command in a new screen tab.

Here is a sample ~/.screenrc file:

escape ^Xx
startup_message off
vbell off
backtick 0 10 10 dfh
backtick 1 10 10 s_uptime
hardstatus alwayslastline "%{.bG}%H%{-}%{.bW} %-w%{.rW}%f%n %t%{-}%+w %=%{..G}[ %l ]%{..Y} %`"
defscrollback 5000
termcapinfo xterm* ti@:te@

In this screenrc file, we note that I run 2 commands every 10 seconds, and that the output of those commands shows up in my status line. dfh simply prints the output of df -h for a certain filesystem, usually /, in a short format, followed by the current time. s_uptime shows load average for the box.

Early on, I used an alias to create new screen windows/tabs. I have switched to using a shell script because I found that when the process terminated (e.g., when the remote server disconnected), the window/tab disappeared without any explanation as to what happened. My script catches that and keeps the tab open until I acknowledge it.

In this script, the variable $MAIN_SCREENS holds the hostnames of all systems on which I want to run screen (this allows this script to function correctly even on systems without screen support), and $SHORT_HOSTNAME is the name of the local system.

Here is the script I use: (the first line is a shebang, but that messes up markdown)

 !/bin/sh
COMMAND="$*"
lines=`echo $MAIN_SCREENS | grep -c $SHORT_HOSTNAME`
if [ $lines -eq 0 ]; then
    # This is not one of the systems running screen, so simply run the command
    ${COMMAND}
else
    # This system runs screen, so proceed as normal

FILENAME=`mktemp ~/.screen_tab.XXXXXX`
cat >${FILENAME} <<EOF
#!/bin/bash

function clean_up {
    # Perform program exit housekeeping
    echo "Command exited with exitcode \${EXITCODE}"
    echo
    echo "Press ENTER to continue..."
    echo
    read
    rm -f ${FILENAME}
    exit
}

trap clean_up SIGINT

${COMMAND}
EXITCODE=\$?
if [ \$EXITCODE -gt 0 ]; then
    clean_up
fi
rm -f ${FILENAME}
EOF
chmod 755 ${FILENAME}
COMMAND=`echo ${COMMAND} | sed 's/.unixdude.net//'`

TERMTYPE=""
if [ ${TERM} = "screen.xterm-256color" ]; then
    TERMTYPE="-T xterm"
fi
screen ${TERMTYPE} -t "${COMMAND}" "${FILENAME}"

fi

Since I always run screen on systems I consider to be jump hosts, I use aliases to connect to those systems. Here is an example:

alias cjb "tabname jump-rdu-01; ssh jump-rdu-01.rdu.domain.com -t 'screen -Rd'; tabname LOCAL_SHELL"

tabname is a short script (it could be an alias) that changes the name of the window/tab in my terminal program: (again, the first line is a shebang)

 !/bin/bash
printf "\e]1;%s\a" $1

A big trick to all of this is reconnecting after a disconnection. My solution here is to use a common, persistent, known value for ${SSH_AUTH_SOCKET}. To do that, I modify SSH_AUTH_SOCKET variable on login. First, I create a symlink called ~/.ssh_auth_socket_${SHORT_HOSTNAME} to the actual $SSH_AUTH_SOCKET, and then I set $SSH_AUTH_SOCKET with a value of ~/.ssh_auth_socket_${SHORT_HOSTNAME}.

The value here is that $SSH_AUTH_SOCKET never changes between logins, so any existing shell sessions, and any newly created ones, all reference the same $SSH_AUTH_SOCKET, and they always work.

setenv SHORT_HOSTNAME `hostname`
find . -maxdepth 1 -name '.ssh_auth_sock*' -mtime +1 -exec rm -f {} \;
find . -maxdepth 1 -name '.screen_tab*' -mtime +1 -exec rm -f {} \;
if $?SSH_AUTH_SOCK then
    if ( `echo $SSH_AUTH_SOCK | grep $SHORT_HOSTNAME | wc -l` == 0 ) then
        setenv NEW_SSH_AUTH_SOCK ~${SUDO_USER}/.ssh_auth_sock_${SHORT_HOSTNAME}
        ln -sf ${SSH_AUTH_SOCK} ${NEW_SSH_AUTH_SOCK}
        setenv SSH_AUTH_SOCK $NEW_SSH_AUTH_SOCK
    endif
endif

This is tcsh code, but bash can do the same thing of course.

macOS Mail issue with postfix

2019-05-14T00:00:00-04:00

I recently got a new MacBook Pro, and as I was setting up the mail configuration on that laptop, I found that I was unable to send mail through my mail server. I use Apple Mail, and usually it works great for me, but in this case it was not working with my Postfix server.

I made several attempts at fixing this issue, but I always got the same error:

postfix/submission/smtpd[70940]: NOQUEUE: reject: RCPT from REDACTED[X.X.X.X]: 554 5.7.1 <recipient@anywhere.com>: Recipient address rejected: Access denied; from=<sender@example.com> to=<recipient@anywhere.com> proto=ESMTP helo=<[REDACTED]>

After exhausting everything I could think of, and after much troubleshooting, I determined that all settings were correct: other systems could send mail through my Postfix server, but I still had the problem with the new laptop.

At one point I had the idea to disable Apple Mail's SMTP server checkbox that sets the application to "Automatically manage connection settings," so I did that and I confirmed the settings that it had been automatically managing -- the port and TLS/SSL setting (587/SSL) and authentication configuration.

Once I disabled the automatic option, everything worked correctly. I didn't actually change any configuration, and all of the settings were correct, it just doesn't work automatically.

I post this that others might find it useful, and that I might find it when I set up my next system.

Scanner

2019-02-20T00:00:00-05:00

I own a CanoScan LiDE 30. This scanner is about 15 years old, and back when I got it new, it was supported on Mac OS X. Canon stopped updating the drivers a couple years later, so I switched to TWAIN-SANE.

When I did that, I wrote a "scan" shell script that uses scanimage to grab a TIFF from the scanner, and then mogrify (part of ImageMagick) to convert the TIFF to JPEG. My scan script saves scans to an NFS automounted directory, so it doesn't matter what computer I scan with, I just happened to do it on my iMac. Or my MacBook Pro - it didn't matter.

Unfortunately, Mattias stopped updating his drivers a few years ago, so I wanted to find another way to use my perfectly good LiDE 30.

I own a Raspberry Pi 3, and I knew that the TWAIN-SANE drivers that Mattias used were simply ports of the open source Linux counterparts, so I installed the SANE and ImageMagick packages on my Raspi and used that for scanning. I used the same scan script that I had on the Mac, and with the NFS automount, everything worked as it had before.

Until this week, that is, when I relocated my Raspi to my living room to connect it to my TV.

Once again, I was looking for a scanning solution. I can't use my Macs (the only computers in my office). My R610 is in my garage, which is not convenient to scanning. I don't want to carry my scanner to the living room to connect to the Raspi, so what can I do?

The answer is easy: I run a single FreeBSD VM on my Synology DS1618+. A quick search, and I discovered sane-backends and sane-fronends are ported to FreeBSD. The question remained: Can my Synology connect a USB page scanner to one of its VMs, and will that work properly? In mere moments, I had the SANE packages and ImageMagick installed on the Synology-hosted FreeBSD VM. And, moments later, I was scanning successfully using exactly the same script I started with on the iMac a decade ago.

Ah, the magic of open source and NFS automounts.

The FreeBSD port has some problems, but it's fine as long as I don't run "scanimage -L" -- that crashes the VM, either a full lockup or a kernel panic and reboot. But, since my scanner works with scanimage, that's all I care about. I just avoid the "-L" parameter, and all is well.

Postfix relay configuration

2019-02-14T00:00:00-05:00

Like most people in my industry, I use multiple email addresses at multiple providers. Used to be, I could configure all of my email addresses in my mail client and just send through my mail server and -- voila -- my email would go out as whatever address I wanted to use.

Then along came Sender Policy Framework (SPF). SPF is a great email-related technology that is designed to protect the integrity of sending domains. Unfortunately for me, adoption of SPF meant that I could no longer send mail through my mail server using one of my other email addresses. Specifically, I could no longer send mail through my server as my gmail.com or mac.com addresses. I couldn't complain, because my server is an origination point for gmail.com or mac.com.

Fortunately, there's a fix for this: Postfix's "sender_dependent_relayhost_maps" feature.

The idea: configure all of my devices to send mail through my mail server, and to have the mail server itself relay as necessary depending on sending address. My goal: send mail directly if it originates from my domain; relay gmail mail through smtp.gmail.com, and relay iCloud mail through smtp.mail.me.com.

To use that, I defined a few settings in my postfix main.cf file:

smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/usr/local/etc/postfix/sender_relay
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd
smtp_tls_note_starttls_offer = yes
smtp_tls_policy_maps = hash:/usr/local/etc/postfix/tls_policy
smtp_tls_security_level = encrypt
smtp_use_tls = yes
smtp_enforce_tls = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_mechanism_filter = plain
smtp_sasl_tls_security_options = noanonymous

For my server, smtp_sender_dependent_authentication is not strictly necessary, because I am the only one sending through my mail server. I include it here because it is needed for a more general solution.

The real magic occurs as a result of sender_dependent_relayhost_maps. This setting configures the different relays based on sending address. My sender_relay file looks like this:

address@gmail.com      [smtp.gmail.com]
address@mac.com        [smtp.mail.me.com]:submission

The smtp_sasl_password_maps configuration is also needed because both Gmail and iCloud require authentication before an email can be relayed through their servers. To do this, I have configured application-specific passwords in both systems, and the sasl_passwd file looks like this:

address@gmail.com               address@gmail.com:PASSWORD
address@mac.com                 address@mac.com:PASSWORD

The TLS policy maps are required because the relays require encryption; the tls_policy file's contents are:

[smtp.gmail.com]:587 encrypt
[smtp.mail.me.com]:587 encrypt

Of course, you must run postmap on each of these files prior to using them.

On FreeBSD, which is my OS of choice, things were slightly more complicated than they otherwise might have been. I was not able to use the default postfix package, because it is not compiled with Cyrus SASL client support. A quick make configure in /usr/ports/mail/postfix allowed me to enable Cyrus SASL client support. After a make and make install I was ready to test. Many thanks to Brad for helping me test.

Now I can eliminate all of the extra accounts on all of my devices.

Network upgrade

2018-12-28T00:00:00-05:00

I have a 3-story house, and my office is on the third floor. Wifi connectivity in the bonus room over the garage was terrible, so I brainstormed ideas with coworkers and friends about how to fix this. As a temporary measure I ran a long ethernet cable from the office to the bonus room, but this had a very low WAF (wife acceptance factor).

Another idea was to try a wifi extender off of my AT&T fiber Internet router. This has higher WAF, but it didn't really seem to help, so I returned it.

A coworker and I came up with a neat idea: I have a conduit running between the 3rd-floor attic space and the garage. What about running a long ethernet cable through that conduit, then moving my router down there? Great idea, but how do I get the Internet signal to the router?

Another coworker had the solution for that: put a managed switch at both ends of the conduit. He just happened to have two spare HP ProCurve 1810-24G switches, so I ran an ethernet cable through the conduit, and voila.

More specifically: I created a VLAN on the switches for the connection between the Fiber/Ethernet transceiver to the router. Then I trunked that VLAN over the conduit-cable. I also trunked my "Internal" VLAN over the same cable to get it back up to the office.

As a bonus, I got to move my noisy R610 to the garage, making my office much quieter. Storage is still in the office, but the server is now remote. It's nice to have iDRAC, now that my server is remote to me.

Now I have excellent wifi service on the first floor and bonus room. So, what about the office and other parts of the second floor? I installed my Synology RT1900ac as a wifi access point in the office; it is now my office switch, and it serves wifi to the parts of the house that the AT&T router does not reach.

testssl.sh

2018-07-02T00:00:00-04:00

Last week, I needed to make some changes to my employer's production website, but before I made those changes in production, I wanted to test them. Unfortunately for me, our dev server is unreachable by Qualys, so I had to come up with another way to test those changes.

Enter testssl.sh. testssl.sh is a shell script that can be used to do testing very simliar to what Qualys does, from a Unix system.

To get started, clone the testssl.sh repo:

git clone https://github.com/drwetter/testssl.sh.git

I was able to make and test the changes I needed on our dev system, before going to production. I was able to confirm that my changes were correct, in a non-production environment.

Many thanks to Dirk Wetter for this excellent tool.

Check it out.

Lab update

2018-06-28T00:00:00-04:00

As mentioned in an earlier post, my ESXi server has been running in a VMware Fusion VM on my iMac. My iMac has 24 GB RAM and 4 logical CPUs; I dediated 10 GB RAM and 2 vCPUs to the ESXi VM.

Last week, I upgraded to a Dell R610 with 96GB RAM and dual Xeon 3 GHz CPUs (16 vCPUs). Although ESXi 6.7 is unsupported on the R610, I installed it and so far it is working great. I am really enjoying the breathing room that so many more CPUs and so much more RAM gives me. Now I can create VMs pretty much as big as I would ever want.

I also made two changes to my PXE configuration.

First, I added a default boot option to the PXE config. Now, it automatically starts a CentOS 7 install after 10 seconds, complete with a kickstart file that adds my user and installs my SSH keys. Mere minutes after I boot a new VM, it is up and running CentOS 7 and I can SSH into it.

Second, I updated my PXE environment to add netboot.xyz. Previously I had the ability to install a half-dozen or so OSes via PXE. Now I can install many more OSes just as easily.

My next update will be to add a managed switch, so that I can use link aggregation for my DS415+ and R610.

I love SSH tunnels

2018-03-15T00:00:00-04:00

Everyone who knows me well knows that tunnels in SSH represent one of my all-time-favorite features, ever. Why? Simple: They are so immensely useful.

Say you are debugging an issue on server, as I am today, and you need a mail server running on that server. But, that server is a production system, and you cannot install a mail server on it. Want a quick "mail server"? Use an SSH tunnel:

sudo ssh -L 25:mail.domain.com:25 daniel@localhost

This will ssh back to localhost as user daniel, but will set up a listener on port 25 that points to mail.domain.com, port 25.

Voila... a temporary "mail server" on your system, torn down when you no longer need it.

You can also use tunnels for many other things. I also use SSH to tunnel VNC, RDP, Synergy, Video Station from my Synology -- and I even use it to access my ESXi console.

SSH tunnels are easy to implement in an SSH config file:

host demo
hostname demo.example.com
user daniel
ForwardAgent yes
LocalForward 5022 192.168.0.2:22
LocalForward 5947 192.168.0.47:5900
LocalForward 24800 192.168.0.2:24800

Home network/home lab

2017-12-03T00:00:00-05:00

I am frequently asked about my home network/home lab setup. The summary is that I currently use a Synology RT1900ac router, and two Synology NAS devices: a DS415+ as primary storage, and a DS214se as backup.

For a number of years, I had a first-generation Drobo; later I added a second generation Drobo S. I liked those, but I wanted something beefier and I wanted to switch from DAS to NAS. I researched the available options, and since I am a ZFS fan, I considered rolling my own server running FreeNAS. I settled on Synology, in large part because of the user interface.

I sold the Drobos and bought a Synology DS415+ NAS, and today I use that for my main storage. My DS415+ is configured with 3x WD Red 3TB and 1x WD Red 4TB, in SHR; this gives about 9TB of usable space. I have a dozen or so NFS exports from the DS415+, and three iSCSI LUNs.

All of my systems, both physical and virtual, are configured for the same user ID, and all Unix systems use the NFS automounter. My iMac also mounts the iSCSI LUNs, for things like iMovie and iPhoto that require block-level storage.

Due to the size of the backup need, I chose a Synology as a backup device. Since this is only for backup, I chose a low-end DS214se; that unit is loaded with 2x WD Red 4TB configured as an 8TB JBOD.

When I first bought the DS415+, I moved data from my iMac's hard drive over to the NAS, but that wasn't really fully utilizing it. I later learned two things that completely changed my world:

First, ESXi can be virtualized inside of VMware Fusion. This was exciting because I did not have money or space for a virtualization lab -- but I did have an iMac with 32GB RAM in it. I prefer ESXi to Fusion for virtualization, but to that point had not been running ESXi at home, and was at that time using Fusion for all of my virtualization needs.
Second, ESXi can use NFS datastores. Fusion requires block-level storage, so I was excited to learn that ESXi can use NFS storage. (At this point, I was very new to ESXi administration.)

Ever since learning these two things about ESXi, I have run an ESXi server at home. I started running ESXi inside a VMware Fusion VM, and today that VM is running ESXi 6.5.0 Update 1. Eventually I would like to migrate this to an Intel NUC, but for now it runs as a VM in VMware Fusion. ESXi sees 10GB of RAM and 2 out of my iMac's 4 CPUs. I connect to my ESXi server two ways: either via Fusion Pro, or via ESXi's built-in web client.

Back to the Synology: The Synology NAS can act as a PXE server, so I have configured pxelinux with a menu that can boot several Linux distros, ESXi, and other things. When I want to install an operating system into a VM, I either use PXE, or I attach an ISO directly to the VM.

Nearly all of my virtualization has now been moved to ESXi. I currently have more than 20 unique OSes installed in VMs in ESX, mostly BSD variants or Linux distros. In VMware Fusion, I run NeXTSTEP 3.3 and OPENSTEP 4.2; these do not seem to run in ESXi, and are the only OSes I run in Fusion anymore. Well, other than ESXi itself, that is...

Once I do buy real hardware for it, the migration will be easy: I will install ESXi on the new system, point it to the existing NFS exports, add the VMs, and start them up.

How to build a Pelican contact form

2017-11-29T00:00:00-05:00

One of the first things I wanted to configure on this site was a contact form. I searched and found that Iain Houston documented how he configured one. I took his ideas and implemented my own slightly different version.

Where he used a separate template, I chose to use the base template and create a Pelican page for my contact form.

The contents of my content/pages/contact.md file are shown here. This results in a page being created at /pages/contact/index.html, which I then added to MENUITEMS.

Title: Contact me
Slug: contact

<div class="col-md-12">
    <header class=jumbotron>
        <h1>Contact me</h1>
        <p>I'd love to hear from you.</p>
    </header>

    <form method="post" action="/php/submit.php">
        <div class="form-group col-md-6">
            <input type="text" class="form-control" id="name"  name="name" placeholder="Name" required>
        </div>
        <div class="form-group col-md-6">
            <input type="email" class="form-control validate email" id="email" name="email" placeholder="Enter email" required>
        </div>
        <p class="antispam">
            Leave this empty:
            <br />
            <input name="url" />
        </p>
        <div class="form-group col-md-12">
          <textarea class="form-control" rows="5" name="message" placeholder="Message" required></textarea>
        </div>
        <div class="form-group col-md-3 offset-md-3">
          <button type="submit" class="btn btn-primary">Submit</button>
        </div>
    </form>
</div>

The content/php/submit.php file is similarly simple:

<?php
if(isset($_POST['url']) && $_POST['url'] == ''){
    //The form was submitted
    $ouremail = 'your@email.com';
    // Important: if we add any form fields to the HTML,
    // and want them included in the email, we will need to add them here also
    $body = "This is the form that was just submitted:
    Name:  $_POST[name]
    E-Mail: $_POST[email]
    Message: $_POST[message]";
    // From:
    // Use the submitters email if they supplied one
    // (and it isn't trying to hack our form).
    // Otherwise send from our email address.
    if( $_POST['email'] && !preg_match( "/[\r\n]/", $_POST['email']) ) {
      $headers = "From: $_POST[email]";
    } else {
      $headers = "From: $ouremail";
    }

    // finally, send the message
    mail($ouremail, 'Contact Form submitted', $body, $headers );
    header('Location: /pages/thank-you/');
}
?>

You will also need a thank-you page. Here's mine, saved as content/pages/thankyou.md:

Title: Thank you for contacting me
slug: thank-you

Your message has been sent.

Iain's script implements simple spam control. To use it properly, you need to add some custom CSS so that the antispam field is hidden from view:

.antispam { display:none;}

Sharing Unix shell config between systems

2017-11-11T00:00:00-05:00

Like most people who work on Unix, I have shell accounts on hundreds of systems. Very few of these systems share a home directory (e.g., via NFS), so I need another way to maintain a common shell configuration between systems.

Based on the design of a former coworker, I have built such a system using git, symlinks, and scripts.

I use four git repos for my Unix account configuration:

a "bin" directory for home
a "bin" directory for work
a "bin" directory for everywhere
an environment directory

All of those directories are private git repos, stored on Bitbucket. I will eventually host my own git repo, but for now I use Atlassian's excellent service.

Of the four repos listed above, the "home" bin directory exists only on personal systems, the "work" directory exists only on employer systems, and the "everywhere" directory is shared on all systems.

The environment directory includes such things as:

tcsh config
bash config (for systems on which tcsh is not available)
ssh config
screenrc
vimrc
a script to configure my environment on a new host

I use three additional scripts in conjunction with the repos:

one to inform me when a sync is required
one to perform the git sync
one to copy an existing config to a new host (such as one that does not have git installed)

Obviously I can run "git push" and "git pull" manually, but since there are four repos here, it was easier to script it.

My "update_repos.sh" script looks like this:

#!/bin/sh

echo "Update my_env"
cd ~/.my_env
git pull
git push

if [ -d ~/bina ]; then
    echo
    echo "Update bina"
    cd ~/bina
    git pull
    git push
fi

if [ -d ~/binh ]; then
    echo
    echo "Update binh"
    cd ~/binh
    git pull
    git push
fi

if [ -d ~/binw ]; then
    echo
    echo "Update binw"
    cd ~/binw
    git pull
    git push
fi

I use symlinks from main files, into my environment directory:

.bash_profile@ -> .my_env/tcsh/bash_profile
.bashrc@ -> .my_env/tcsh/bashrc
.cshrc@ -> .my_env/tcsh/cshrc
.login@ -> .my_env/tcsh/login
.logout@ -> .my_env/tcsh/logout
.ssh@ -> .my_env/ssh
.vimrc@ -> .my_env/vimrc

To bootstrap a host, I check out the applicable repos on the host, then run my setup script, which looks like this:

#!/bin/sh
ln -sf .my_env/tcsh/bashrc     ~/.bashrc
ln -sf .my_env/tcsh/cshrc      ~/.cshrc
ln -sf .my_env/tcsh/login      ~/.login
ln -sf .my_env/tcsh/logout     ~/.logout
ln -sf .my_env/vimrc           ~/.vimrc
ln -sf .my_env/screenrc        ~/.screenrc
rm ~/.ssh
ln -s  .my_env/ssh             ~/.ssh
if [ -e '/auto/downloads/Set up new host/ssh_config/id_rsa' ]; then
    cp '/auto/downloads/Set up new host/ssh_config/id_rsa' ~/.ssh
fi

Two comments about this script:

~/.ssh is not removed if it is a directory. This does complicate matters on systems that already have a ~/.ssh directory. In those cases I blow away ~/.ssh and rebuild it as a link.
/auto is where my NFS automounter lives. /auto/downloads is on my NAS and in keeping with best-practices my private key only exists on appropriate systems.

Comment below if you have any questions or comments about this setup.

Welcome

2017-10-26T10:21:00-04:00

A while ago, a friend suggested that I should start a blog. Around that same time, a coworker introduced me to DigitalOcean. I quickly set up a VM at DigitalOcean (that VM is serving this web page), and installed FreeBSD 11.

I looked at WordPress and other popular blogging tools, but all were overkill for what I want to do, and all would overtax the small VM I am running. I finally settled on Pelican, after a coworker showed me his blog.

I played around with all of the Pelican themes, and wanted one that would look good on all screen sizes, so it had to be a Bootstrap derivative.

I use a second FreeBSD system for the development version of this blog. That system is a VM running on my ESXi server. I develop and deploy there, then I deploy the production version here via rsync.

I won't get into the details of everything this VM does, but it functions as a:

Mail server, running Postfix + Dovecot + SpamAssassin + Sieve + Roundcube
Web server, running Apache serving a dozen or so virtual hosts, all of which use Let's Encrypt certificates