Today, I spent half my day debugging an email error. I got a lot of bounces saying:
552 5.2.0 Message contains bare CR and is violating 822.bis section 2.3 (in reply to end of DATA command))
In the 20+ years I'm running mailservers, I had never heard of this error before. And Google couldn't make me much wiser either.
The mailserver generating the message was mxin5.ziggo.nl - the primary server for Dutch provider Ziggo. I could not figure out which MTA they are using, although the banner right after an EHLO looks like Sendmail. I downloaded the latest Sendmail sources, but this error message does not come from a vanilla Sendmail installation. Other software like FortiMail may be a Sendmail fork or mimic the Sendmail signatures. But of course we cannot examine their source code.
I did find some references that qmail may have problems with bare LFs. But those were with regards to LF (line feed), not CR (carriage return). Downloaded the source of qmail and netqmail to find that qmail was probably not the MTA used either.
Then I tried to trace my own postfix server. Let's see that the mail does indeed contain a bare CR. I did not believe there was a bare CR as I examined the sourcecode of the generating software. Tracing is a lot harder nowadays since starttls will prevent a simple tcpdump. As an alternative, I used postfix' header_checks to HOLD a suspected mail and sure enough, the software generating the message included a stray CR = hex char %0D = \r in a small percentage of outgoing mails. The generating software used an external data source which inserted the CR.
The bug generating the CR in the emails is years old, the mails were always accepted by Ziggo without any problem. My best guess is that Ziggo recently upgraded their mailserver and the new version now forbids the CR.
Last week, Red Hat published their latest and greatest Fedora version, number 39. That means Fedora 38 will still be supported for about half a year. Fedora 37 even has nog more than a few more weeks of support left. So over the weekend, I updated my laptop. First I backupped my whole system to a USB stick and then updated using the well-known formula:
Updating was a pretty smooth operation. But today, when I wanted to login to my servers in the datacenter, I found that OpenVPN could not connect anymore. While the configuration file hadn't changed even one letter. Time to investigate.
Problem 1: Ciphers
First, I found this message in the logs:
DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations.
The --data-ciphers option is something new to OpenVPN 2.5 and in version 2.6 it defaults to AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305. Because AES-256-CBC is not in there, OpenVPN cannot setup encryption anymore. To fix this, I added:
WARNING: 'tun-mtu' is used inconsistently, local='tun-mtu 1532', remote='tun-mtu 1500' WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1590', remote='link-mtu 1622' Option inconsistency warnings triggering disconnect due to --opt-verify SENT CONTROL [mekker]: 'AUTH_FAILED' (status=1)
I've seen both warning-messages combined, but also each separate, with dozens of different numbers. And always they were followed by a "opt-verify" and a AUTH_FAILED message. For me, the big solution was to fiddle a bit with the --tun-mtu option. Please note that the signature of this option used to be:
--tun-mtu n = Take the TUN device MTU to be n and derive the link MTU from it (default=1500).
But is now:
--tun-mtu args = tun-mtu [tun-mtu] [occ-mtu]
In my specific situation, I got all connections working using the following settings:
Server side Version
Server side Settings
Client side Version
Client side Settings
OpenVPN 2.4.7 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 20 2019 library versions: OpenSSL 1.1.1d 10 Sep 2019, LZO 2.10
The crypto-policy that comes with Red Hat Enterprise Linux 9 and AlmaLinux 9 does not allow RSA key sizes below 2048 bit. Some legacy SSH servers are configured with 1024 bit key sizes though which will result in connection failures:
# ssh username@servername.example.com
Bad server host key: Invalid key length
Trying to find the issue using ssh -vvv, you would get:
OpenSSH_8.8p1, OpenSSL 3.0.5 5 Jul 2022
debug1: Reading configuration data /home/geeklab/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/geeklab/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-jxs.conf
debug1: Connecting to servername.example.com [1.2.3.4] port 22.
debug1: Connection established.
debug1: identity file /home/geeklab/.ssh/id_rsa type 0
debug1: identity file /home/geeklab/.ssh/id_rsa-cert type -1
debug1: identity file /home/geeklab/.ssh/id_dsa type -1
debug1: identity file /home/geeklab/.ssh/id_dsa-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ed25519 type 3
debug1: identity file /home/geeklab/.ssh/id_ed25519-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ed25519_sk type -1
debug1: identity file /home/geeklab/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/geeklab/.ssh/id_xmss type -1
debug1: identity file /home/geeklab/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: compat_banner: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000002
debug1: Authenticating to servername.example.com:22 as 'root'
debug1: load_hostkeys: fopen /home/geeklab/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32
debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_GROUP received
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: SSH2_MSG_KEX_DH_GEX_REPLY received
Bad server host key: Invalid key length
While the best solution would be to fix the key size of the target server, this is not always possible on embedded devices. For those devices, you could alternatively make a change to your ~/.ssh/config file:
Host servername.example.com
RSAMinSize 1024
In response to Kukulkan's comment, I checked out the source code for the OpenSSH 9.8p1 that I'm running. I found these changelog items:
This and a quick inspection of the source code leads my to believe that originally RSAMinSize was added by CentOS, then imported into the RedHat code. After that, upstream added an officially supported variable named RequiredRSASize, but RSAMinSize was still maintained for compatibility. (On Fedora 41 both options work fine.)
However, distros that never had the RSAMinSize patch to begin with, still don't support it. If they're recent enough, they should support RequiredRSASize though.
One of my customers is still using a very old library for iDeal online payments.
It's old, but it's also really stable. Or rather, was stable. Until today. Today I tried to switch that customer from ideal.secure-ing.com to the new ideal-acquiring.ing.nl. That didn't go so well.
The switch itself was pretty simple. Change the merchant ID, do a little chair dance with the SSL certificates and change the hostname. Unfortunately, that turned out not to be the full fix.
ING Bank kept returning this very useful error message "System generating error: Acquirer":
<errorCode>SO1000</errorCode><errorMessage>Failure in system</errorMessage><errorDetail>System generating error: Acquirer</errorDetail>
According to the iDeal manual, the Acquirer is the bank of the Merchant (=webshop). So when the website I'm working on is connecting to their bank, the bank basically says in their message that they themselves are having a technical issue. For several hours, both on the live environment and on sandbox environment. Sounds a bit unbelievable to me.
As a external consultant I cannot call the bank for help, I'm condemned to use public resources and spend a lot of time to find the issue here.
Took me several hours of trying to change certificates, changing the Merchant ID and other parameters, but nothing helped. Until I noticed the library this customer is using opens it's own connection to the bank:
Could it be that the new environment requires different/more secure SSL/TLS? I did a tcpdump on the network and opened the dump in wireshark. The connection was using TLSv1.2, nothing to worry about.
Hey, that's weird. HTTP/1.0, did we not abandon that protocol 20 years ago? Could it be that using an old protocol blocks the whole transaction? Turns out I was right. I replaced the whole function to use curl, and the problem was instantly fixed:
That's it. For now. At some moment I'll probably advise this customer to change to a more modern iDeal lib.
I'm a bit annoyed though. The bank could give an answer that's more indicative of the problem. How hard is it for the ING Bank to always include a reason if the request gets past the webserver to begin with? The error message was digitally signed with a SSL-based XML signature, which I think means the backend application could have done better.
It has been annoying a lot of people for a long long time: Ubiquiti's unwillingness to fully support multi-VLAN networks on all of its products. And their constant push towards "the cloud", a nice marketing term for "another persons computer". Ubiquiti has a great set of products. But if you like to stay away from the cloud and don't want to create a ui.com account, some functions are hard to get to work or just don't work at all.
This is especially annoying with the CloudKey Plus, a device that runs the management software for your network AND a NVR (Network Video Recorder) named "UniFi Protect" at the same time. You should always isolate network management from normal users, but in the case of the CloudKey Plus that also limits access to Protect.
"With the UDM-Pro, can I assign the applications their own IP address? For example, can the UDM-Pro/network app be on a different subnet than the Unifi Protect app? I prefer to keep my network devices on a different subnets than my cameras."
"The new feature of Offline Access from the mobile apps is great, but in our case we have some separate networks for wireless, protect, computers and several remote sites. It doesn't make us sense that the direct connection works only on the direct subnet having routing enabled between the subnets. [..] From the security point of view, it could expose a 'risk' having a phone beaconing the IPs of the protect devices, but I think that can be acceptable or have switch to enable/disable the feature on the NVR Unifi OS."
When someone else mentions the topic is so silent, _Space responds:
"I dont think the lack of reply is due to people having flat networks, I think it is many people just giving up.
As they say at work, no horse is too dead to beat... but this topic has been beaten to death... People BEG for the ability to use protect from a mobile device, no cloud and point to an IP. maybe port forwarding, maybe vpn, maybe just on the local LAN... lots of scenarios.
UI is aware of what people are begging for.
BUT, UI has a plan. they wont tell anyone what it is and the are determined to keep their cloud in the mix."
Let's see what we can accomplish.
My network setup
For this article, my network will be set up as follows:
EdgeRouter 12
Multiple UniFi switches
CloudKey Plus (UCKP)
G4 Bullet camera
VLAN4 = 192.168.4.0/24 = management VLAN that contains the camera but also the CloudKey Plus (UCKP). This VLAN is on switch0.4 of my EdgeRouter.
VLAN5 = 192.168.5.0/24 = LAN, where my laptop and phone are connected. This VLAN is on switch0.5.
Failed attempts to get the Unifi Protect app to connect to the console
The Ubiquiti Protect app scans the local network by sending out a broadcast to 255.255.255.255 on udp port 10001. If there only was a way to relay that message to another VLAN, the app might be able to detect the Protect appliance.
First I looked to see it my EdgeRouter 12 supports forwarding of broadcast packets, but I haven't found a built-in way to do that. In Odi's astoundingly incomplete notes I found an interesting alternative, using iptables' TEE target. The command to run would be:
iptables -t mangle -A INPUT -i switch0.5 -d 255.255.255.255 -j TEE --gateway 192.168.4.255
Unfortunately, my EdgeRouter does have the libxt_TEE.so required for this, but lacks the corresponding kernel modules to make this target work.
I then found a post that claims to have a way to use iptables' PREROUTING redirect a packet for 255.255.255.255 to a specific address on another VLAN. According to that page, I should be able to route Ubiquiti's discovery packets to my CloudKey using the following command:
I don't see anything wrong with this, but still it does not work. I'm not sure, but I think the global broadcast address 255.255.255.255 may be explicitly excluded from forwarding by the Linux network stack.
Another suggestion I found, was to install ubnt-bcast-relay which is based on udp-broadcast-relay. By installing this software on your edgerouter, you can easily relay packets from one network to another. This may be a useful method for some people, but personally I don't feel like running third-party software on my router. Also, I'm not sure this will work as the EdgeRouter itself is listening on udp port 10001.
How I connected my app to the Unifi Protect server
I could not find a satisfying software-based solution for this issue. The best option would be for Ubiquiti to just add a text field in which you can type an IP, but they won't. Setting up a temporary Network Bridge between VLAN4 and VLAN5 could help the app to connect to the Protect NVR, and then remove the bridge again. This is a bit risky, because adding the interface you're connected to to a bridge will make you lose your connection.
Instead, I chose to set up a "hardware bridge": a simple network cable. On one of my switches, I picked two unused ports, set one to VLAN4, the other to VLAN5. Disabling Spanning Tree Protocol for the ports is a requirement, or the ports will quickly go into blocking mode. Then I connected a UTP cable between them, letting all traffic flow between VLAN4 and VLAN5. I set up the protect app and 30 seconds later I removed the cable and reversed the settings.
Yay! We got a working app! Or do we....?
We got the app set up, but I doubt we actually got a long lasting solution. At least not if you're trying to get your Unifi Protect to work over a VPN. Even now we can connect cross-VLAN or over a VPN, the app is flaky at best.
The very first time I tried to access my camera's via VPN over a 4G connection, the Protect App began to stutter. And complain about how "Some VPN networks may cause connectivity issues".
Yes, some VPN networks may do that. But I manage my own, and it's set up well. Why does Ubiquiti actively try to scare people away from VPNs?
Right after the message above, the connection is magically lost. And then the UCKP is grayed out and becomes unusable.
Unfortunately for Ubiquiti a few tools let us see right through their bullshit. Using any browser on the phone, we can still connect to the CloudKey just fine. Also, tcpdump shows traffic is just fine. The app sometimes works for a short while, then without any good reason loses connection to the UCKP. And there seems no acceptable method to get it back to work.
Once the UCKP is grayed out, the only way to get it back, is to log off the whole app and re-discover the CloudKey. Not even reconnecting to the original wifi gets it back online if VLAN4 and VLAN5 are not directly linked to each other.
It seems that the app constantly wants to keep re-discovering the UCKP, and if it can't, it will stop functioning.
My suggestion is to stay away from Ubiquiti camera's.
After hours of trying, I'm done with it. I have deleted the UniFi Protect app from my phone. And I'm returning the €190 camera to the webshop. For my office, I'll probably end up with another brand that does respects its customers' wishes.
I did not test this, but I figured it could be useful.
A dirty solution for discovery
A dirty solution that could help out, is to use socat to tunnel the connection. Then you'd have to change the source ip as well. I just came up with this solution, but have not tested it yet:
A Hexadecimal color code looks like #ff7f00. This means:
red: ff hexadecimal = 255 decimal
green: 7f hexadecimal = 127 decimal
blue: 00 hexadecimal = 0 decimal
And that will generate a orange color with hex value ff7f00.
One can think of several very difficult methods to generate 3 numbers between 0 and 255, then concatenating their hexadecimal values. But there's a much easier way.
Since the string is formatted as a 6 character string, the value of which is between 000000 and ffffff (hex), you could just pick any random value between 0 and 16777215 (=hex ffffff) and convert that to a hex string.
Both of these lines will generate a random Hex Color Code:
While installing Microsoft SQL Server on a VMWare, I had this error:
"This SQL Server setup media does not support the language of the OS, or does not have the SQL Server English language version installation files. Use the matching language-specific SQL Server media, or install both the language specific MUI and change the format and system locales through the regional settings in the control panel."
I found a lot of possible solutions, especially those on https://www.ryadel.com/en/sql-server-2017-setup-install-error-oops-language-not-supported-rules-fix/ looked promising. But they did not work in the end.
The solution turned out to be mounting the iso file locally on the target machine. SQL Server comes as a .iso file. If you mount that via VMWare CDROM emulation or via a network share, you'll run into problems installing. But copy the iso to the target machine and double click it, then you can run setup.exe without problems.
Loading...
