Here is a build of Swift 5.3 for the Raspberry Pi 4 running Ubuntu Mate 20.04. See my post about the Odroid C2 for installation instructions.

USE AT YOUR OWN RISK!!! I provide absolutely no warrantee with this at all!

Swift 5.3 for the Raspberry Pi 4: https://www.dropbox.com/s/7l39d5df1zvbkm8/swift5.3-aarch64-rpi4-ubuntu-linux-20.04.tar.bz2?dl=0