552 5.2.0 Message contains bare CR and is violating 822.bis section 2.3 (in reply to end of DATA command))

Today, I spent half my day debugging an email error. I got a lot of bounces saying:

552 5.2.0 Message contains bare CR and is violating 822.bis section 2.3 (in reply to end of DATA command))

In the 20+ years I'm running mailservers, I had never heard of this error before. And Google couldn't make me much wiser either.

The mailserver generating the message was mxin5.ziggo.nl - the primary server for Dutch provider Ziggo. I could not figure out which MTA they are using, although the banner right after an EHLO looks like Sendmail. I downloaded the latest Sendmail sources, but this error message does not come from a vanilla Sendmail installation. Other software like FortiMail may be a Sendmail fork or mimic the Sendmail signatures. But of course we cannot examine their source code.

I did find some references that qmail may have problems with bare LFs. But those were with regards to LF (line feed), not CR (carriage return). Downloaded the source of qmail and netqmail to find that qmail was probably not the MTA used either.

Then I tried to trace my own postfix server. Let's see that the mail does indeed contain a bare CR. I did not believe there was a bare CR as I examined the sourcecode of the generating software. Tracing is a lot harder nowadays since starttls will prevent a simple tcpdump. As an alternative, I used postfix' header_checks to HOLD a suspected mail and sure enough, the software generating the message included a stray CR = hex char %0D = \r in a small percentage of outgoing mails. The generating software used an external data source which inserted the CR.

The bug generating the CR in the emails is years old, the mails were always accepted by Ziggo without any problem. My best guess is that Ziggo recently upgraded their mailserver and the new version now forbids the CR.

© GeekLabInfo 552 5.2.0 Message contains bare CR and is violating 822.bis section 2.3 (in reply to end of DATA command)) is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)
Loading...

OpenVPN 2.6.6 on Fedora 39: How to fix your connectivity to legacy OpenVPN servers!

Last week, Red Hat published their latest and greatest Fedora version, number 39. That means Fedora 38 will still be supported for about half a year. Fedora 37 even has nog more than a few more weeks of support left. So over the weekend, I updated my laptop. First I backupped my whole system to a USB stick and then updated using the well-known formula:

sudo dnf-y update && \
sudo dnf -y install dnf-plugin-system-upgrade && \
sudo dnf system-upgrade download –releasever=39 && \
sudo dnf system-upgrade reboot

Updating was a pretty smooth operation. But today, when I wanted to login to my servers in the datacenter, I found that OpenVPN could not connect anymore. While the configuration file hadn't changed even one letter. Time to investigate.

Problem 1: Ciphers

First, I found this message in the logs:

DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305).
OpenVPN ignores --cipher for cipher negotiations.

The --data-ciphers option is something new to OpenVPN 2.5 and in version 2.6 it defaults to AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305. Because AES-256-CBC is not in there, OpenVPN cannot setup encryption anymore. To fix this, I added:

--data-ciphers AES-256-CBC:AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305

Problem 2: MTU

Then I found a message:

WARNING: 'tun-mtu' is used inconsistently, local='tun-mtu 1532', remote='tun-mtu 1500'
WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1590', remote='link-mtu 1622'
Option inconsistency warnings triggering disconnect due to --opt-verify
SENT CONTROL [mekker]: 'AUTH_FAILED' (status=1)

I've seen both warning-messages combined, but also each separate, with dozens of different numbers. And always they were followed by a "opt-verify" and a AUTH_FAILED message. For me, the big solution was to fiddle a bit with the --tun-mtu option. Please note that the signature of this option used to be:

--tun-mtu n = Take the TUN device MTU to be n and derive the link MTU from it (default=1500).

But is now:

--tun-mtu args = tun-mtu [tun-mtu] [occ-mtu]

In my specific situation, I got all connections working using the following settings:

Server side VersionServer side SettingsClient side VersionClient side Settings
OpenVPN 2.4.7 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 20 2019
library versions: OpenSSL 1.1.1d 10 Sep 2019, LZO 2.10

Running on a Raspbian 10.2 device
no mtu settings, just defaultsOpenVPN 2.6.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
--tun-mtu 1500 1532
OpenVPN 2.5.9 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 16 2023
library versions: OpenSSL 3.0.7 1 Nov 2022, LZO 2.10

Running on AlmaLinux 9.2
no mtu settings, just defaultsOpenVPN 2.6.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
--tun-mtu 1500 1532
OpenVPN 2.4.9 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 24 2020
library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.03

Running on CentOS 6.10
no mtu settings, just defaultsOpenVPN 2.6.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
--tun-mtu 1500 1500
OpenVPN 2.4.7 mips-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 22 2022
library versions: OpenSSL 1.0.2u 20 Dec 2019, LZO 2.08

Running on a Ubiquiti EdgeRouter 12P
no mtu settings, just defaultsOpenVPN 2.6.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
[yet to be tested]

© GeekLabInfo OpenVPN 2.6.6 on Fedora 39: How to fix your connectivity to legacy OpenVPN servers! is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

Allow RSA key size below 2048 bit to successfully establish SSH connections to legacy systems

The crypto-policy that comes with Red Hat Enterprise Linux 9 and AlmaLinux 9 does not allow RSA key sizes below 2048 bit. Some legacy SSH servers are configured with 1024 bit key sizes though which will result in connection failures:

# ssh username@servername.example.com
Bad server host key: Invalid key length

Trying to find the issue using ssh -vvv, you would get:

OpenSSH_8.8p1, OpenSSL 3.0.5 5 Jul 2022
debug1: Reading configuration data /home/geeklab/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/geeklab/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/99-jxs.conf
debug1: Connecting to servername.example.com [1.2.3.4] port 22.
debug1: Connection established.
debug1: identity file /home/geeklab/.ssh/id_rsa type 0
debug1: identity file /home/geeklab/.ssh/id_rsa-cert type -1
debug1: identity file /home/geeklab/.ssh/id_dsa type -1
debug1: identity file /home/geeklab/.ssh/id_dsa-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/geeklab/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ed25519 type 3
debug1: identity file /home/geeklab/.ssh/id_ed25519-cert type -1
debug1: identity file /home/geeklab/.ssh/id_ed25519_sk type -1
debug1: identity file /home/geeklab/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/geeklab/.ssh/id_xmss type -1
debug1: identity file /home/geeklab/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: compat_banner: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000002
debug1: Authenticating to servername.example.com:22 as 'root'
debug1: load_hostkeys: fopen /home/geeklab/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32
debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_GROUP received
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: SSH2_MSG_KEX_DH_GEX_REPLY received
Bad server host key: Invalid key length

While the best solution would be to fix the key size of the target server, this is not always possible on embedded devices. For those devices, you could alternatively make a change to your ~/.ssh/config file:

Host servername.example.com
     RSAMinSize 1024
© GeekLabInfo Allow RSA key size below 2048 bit to successfully establish SSH connections to legacy systems is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

iDeal error SO1000

One of my customers is still using a very old library for iDeal online payments.

It's old, but it's also really stable. Or rather, was stable. Until today. Today I tried to switch that customer from ideal.secure-ing.com to the new ideal-acquiring.ing.nl. That didn't go so well.

The switch itself was pretty simple. Change the merchant ID, do a little chair dance with the SSL certificates and change the hostname. Unfortunately, that turned out not to be the full fix.

ING Bank kept returning this very useful error message "System generating error: Acquirer":

<errorCode>SO1000</errorCode><errorMessage>Failure in system</errorMessage><errorDetail>System generating error: Acquirer</errorDetail>
screenshot of xml error message

According to the iDeal manual, the Acquirer is the bank of the Merchant (=webshop). So when the website I'm working on is connecting to their bank, the bank basically says in their message that they themselves are having a technical issue. For several hours, both on the live environment and on sandbox environment. Sounds a bit unbelievable to me.

As a external consultant I cannot call the bank for help, I'm condemned to use public resources and spend a lot of time to find the issue here.

Took me several hours of trying to change certificates, changing the Merchant ID and other parameters, but nothing helped. Until I noticed the library this customer is using opens it's own connection to the bank:

<?php $f=fsockopen("ssl://ideal-acquiring.ing.nl", ...);

Could it be that the new environment requires different/more secure SSL/TLS? I did a tcpdump on the network and opened the dump in wireshark. The connection was using TLSv1.2, nothing to worry about.

Lets see what else this code does:

$out.='POST ' . $path . ' HTTP/1.0' . $this->CRLF;

Hey, that's weird. HTTP/1.0, did we not abandon that protocol 20 years ago? Could it be that using an old protocol blocks the whole transaction? Turns out I was right. I replaced the whole function to use curl, and the problem was instantly fixed:

protected function postToHost($url, $data, $timeout = 30 /*ignored now*/ ) {
            $url=preg_replace("#^ssl://#","https://",$url);
            $ch = curl_init($url);
            curl_setopt($ch, CURLOPT_POST, 1);
            curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
            curl_setopt($ch, CURLOPT_POSTFIELDS, "$data");
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
            $output = curl_exec($ch);
            return $output;
}

That's it. For now. At some moment I'll probably advise this customer to change to a more modern iDeal lib.

I'm a bit annoyed though. The bank could give an answer that's more indicative of the problem. How hard is it for the ING Bank to always include a reason if the request gets past the webserver to begin with? The error message was digitally signed with a SSL-based XML signature, which I think means the backend application could have done better.

© GeekLabInfo iDeal error SO1000 is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5.00 out of 5)
Loading...

Connect your UniFi Protect app to a console in a different VLAN or over a VPN

It has been annoying a lot of people for a long long time: Ubiquiti's unwillingness to fully support multi-VLAN networks on all of its products. And their constant push towards "the cloud", a nice marketing term for "another persons computer". Ubiquiti has a great set of products. But if you like to stay away from the cloud and don't want to create a ui.com account, some functions are hard to get to work or just don't work at all.

This is especially annoying with the CloudKey Plus, a device that runs the management software for your network AND a NVR (Network Video Recorder) named "UniFi Protect" at the same time. You should always isolate network management from normal users, but in the case of the CloudKey Plus that also limits access to Protect.

Some of the complaints of Ubiquiti users

As alawley posts on the community forums:

"With the UDM-Pro, can I assign the applications their own IP address? For example, can the UDM-Pro/network app be on a different subnet than the Unifi Protect app? I prefer to keep my network devices on a different subnets than my cameras."

In the thread [Feature Request] Direct access to Unifi Protect from different subnet, AlexRoig says:

"The new feature of Offline Access from the mobile apps is great, but in our case we have some separate networks for wireless, protect, computers and several remote sites. It doesn't make us sense that the direct connection works only on the direct subnet having routing enabled between the subnets.
[..]
From the security point of view, it could expose a 'risk' having a phone beaconing the IPs of the protect devices, but I think that can be acceptable or have switch to enable/disable the feature on the NVR Unifi OS."

When someone else mentions the topic is so silent, _Space responds:

"I dont think the lack of reply is due to people having flat networks, I think it is many people just giving up.

As they say at work, no horse is too dead to beat... but this topic has been beaten to death... People BEG for the ability to use protect from a mobile device, no cloud and point to an IP. maybe port forwarding, maybe vpn, maybe just on the local LAN... lots of scenarios.

UI is aware of what people are begging for.

BUT, UI has a plan. they wont tell anyone what it is and the are determined to keep their cloud in the mix."

Let's see what we can accomplish.

My network setup

For this article, my network will be set up as follows:

  • EdgeRouter 12
  • Multiple UniFi switches
  • CloudKey Plus (UCKP)
  • G4 Bullet camera
  • VLAN4 = 192.168.4.0/24 = management VLAN that contains the camera but also the CloudKey Plus (UCKP). This VLAN is on switch0.4 of my EdgeRouter.
  • VLAN5 = 192.168.5.0/24 = LAN, where my laptop and phone are connected. This VLAN is on switch0.5.

Failed attempts to get the Unifi Protect app to connect to the console

The Ubiquiti Protect app scans the local network by sending out a broadcast to 255.255.255.255 on udp port 10001. If there only was a way to relay that message to another VLAN, the app might be able to detect the Protect appliance.

First I looked to see it my EdgeRouter 12 supports forwarding of broadcast packets, but I haven't found a built-in way to do that. In Odi's astoundingly incomplete notes I found an interesting alternative, using iptables' TEE target. The command to run would be:

iptables -t mangle -A INPUT -i switch0.5 -d 255.255.255.255 -j TEE --gateway 192.168.4.255

Unfortunately, my EdgeRouter does have the libxt_TEE.so required for this, but lacks the corresponding kernel modules to make this target work.

I then found a post that claims to have a way to use iptables' PREROUTING redirect a packet for 255.255.255.255 to a specific address on another VLAN. According to that page, I should be able to route Ubiquiti's discovery packets to my CloudKey using the following command:

iptables -t nat -A PREROUTING -i switch0.5 -m udp -p udp -d 255.255.255.255 --dport 10001 -j DNAT --to 192.168.4.253
iptables -I FORWARD -m udp -p udp --dport 10001 -j ACCEPT

I don't see anything wrong with this, but still it does not work. I'm not sure, but I think the global broadcast address 255.255.255.255 may be explicitly excluded from forwarding by the Linux network stack.

Another suggestion I found, was to install ubnt-bcast-relay which is based on udp-broadcast-relay. By installing this software on your edgerouter, you can easily relay packets from one network to another. This may be a useful method for some people, but personally I don't feel like running third-party software on my router. Also, I'm not sure this will work as the EdgeRouter itself is listening on udp port 10001.

How I connected my app to the Unifi Protect server

I could not find a satisfying software-based solution for this issue. The best option would be for Ubiquiti to just add a text field in which you can type an IP, but they won't. Setting up a temporary Network Bridge between VLAN4 and VLAN5 could help the app to connect to the Protect NVR, and then remove the bridge again. This is a bit risky, because adding the interface you're connected to to a bridge will make you lose your connection.

Instead, I chose to set up a "hardware bridge": a simple network cable. On one of my switches, I picked two unused ports, set one to VLAN4, the other to VLAN5. Disabling Spanning Tree Protocol for the ports is a requirement, or the ports will quickly go into blocking mode. Then I connected a UTP cable between them, letting all traffic flow between VLAN4 and VLAN5. I set up the protect app and 30 seconds later I removed the cable and reversed the settings.

Yay! We got a working app! Or do we....?

We got the app set up, but I doubt we actually got a long lasting solution. At least not if you're trying to get your Unifi Protect to work over a VPN. Even now we can connect cross-VLAN or over a VPN, the app is flaky at best.

The very first time I tried to access my camera's via VPN over a 4G connection, the Protect App began to stutter. And complain about how "Some VPN networks may cause connectivity issues".

Yes, some VPN networks may do that. But I manage my own, and it's set up well. Why does Ubiquiti actively try to scare people away from VPNs?

Right after the message above, the connection is magically lost. And then the UCKP is grayed out and becomes unusable.

Unfortunately for Ubiquiti a few tools let us see right through their bullshit. Using any browser on the phone, we can still connect to the CloudKey just fine. Also, tcpdump shows traffic is just fine. The app sometimes works for a short while, then without any good reason loses connection to the UCKP. And there seems no acceptable method to get it back to work.

Once the UCKP is grayed out, the only way to get it back, is to log off the whole app and re-discover the CloudKey. Not even reconnecting to the original wifi gets it back online if VLAN4 and VLAN5 are not directly linked to each other.

It seems that the app constantly wants to keep re-discovering the UCKP, and if it can't, it will stop functioning.

My suggestion is to stay away from Ubiquiti camera's.

After hours of trying, I'm done with it. I have deleted the UniFi Protect app from my phone. And I'm returning the €190 camera to the webshop. For my office, I'll probably end up with another brand that does respects its customers' wishes.

Alternative solutions for discovery

The following solution could also be interesting:

https://github.com/bahamas10/unifi-proxy - A NodeJS program that relays broadcast messages

I did not test this, but I figured it could be useful.

A dirty solution for discovery

A dirty solution that could help out, is to use socat to tunnel the connection. Then you'd have to change the source ip as well. I just came up with this solution, but have not tested it yet:

On a Linux computer with IP 192.168.4.1 run:

socat -u tcp-listen:50000 udp-sendto:255.255.255.255:10001,broadcast
iptables -t nat -I POSTROUTING -d 255.255.255.255 -j SNAT --to-source 192.168.5.1

Then on a Linux computer with IP 192.168.5.1 run:

socat -u udp-recvfrom:10001,fork tcp:192.168.4.1:50000

© GeekLabInfo Connect your UniFi Protect app to a console in a different VLAN or over a VPN is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)
Loading...

How to generate a random Hex Color Code in Javascript

A Hexadecimal color code looks like #ff7f00. This means:

  • red: ff hexadecimal = 255 decimal
  • green: 7f hexadecimal = 127 decimal
  • blue: 00 hexadecimal = 0 decimal

And that will generate a orange color with hex value ff7f00.

One can think of several very difficult methods to generate 3 numbers between 0 and 255, then concatenating their hexadecimal values. But there's a much easier way.

Since the string is formatted as a 6 character string, the value of which is between 000000 and ffffff (hex), you could just pick any random value between 0 and 16777215 (=hex ffffff) and convert that to a hex string.

Both of these lines will generate a random Hex Color Code:

'#' + Math.floor(Math.random()*0xffffff).toString(16);
'#' + Math.floor(Math.random()*16777215).toString(16);

Simple huh?

© GeekLabInfo How to generate a random Hex Color Code in Javascript is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

“SQL Server setup does not support the language of the OS…” My solution

While installing Microsoft SQL Server on a VMWare, I had this error:

"This SQL Server setup media does not support the language of the OS, or does not have the SQL Server English language version installation files. Use the matching language-specific SQL Server media, or install both the language specific MUI and change the format and system locales through the regional settings in the control panel."

I found a lot of possible solutions, especially those on https://www.ryadel.com/en/sql-server-2017-setup-install-error-oops-language-not-supported-rules-fix/ looked promising. But they did not work in the end.

The solution turned out to be mounting the iso file locally on the target machine. SQL Server comes as a .iso file. If you mount that via VMWare CDROM emulation or via a network share, you'll run into problems installing. But copy the iso to the target machine and double click it, then you can run setup.exe without problems.

© GeekLabInfo "SQL Server setup does not support the language of the OS..." My solution is a post from GeekLab.info. You are free to copy materials from GeekLab.info, but you are required to link back to http://www.geeklab.info

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...