This HowTo is mainly meant as documentation if I have to resetup my lab 🙂

Hardware

Ryzen 7 4750G
64GB RAM
Network Interface card with 4 ports
1TB NVME M2 disk

Prerequisites

Internet Connection (in this case via my pfSense VM running on old lab)
Backup of pfSense config

Setup

After doing a minimal Fedora 32 Server installation change to root-user and change hostname to something pretty to show in console
```
hostnamectl set-hostname homeserver
```

Install some stuff needed later and do an update after that

dnf install vim-enhanced virt-manager libvirt-client NetworkManager-ovs xorg-x11-server-Xorg xorg-x11-xauth

Add to .bashrc

alias vi='vim'

# to resolv "tput: No value for $TERM and no -T specified" messages if you use scp for example (checks if session is interactive, if not the PS1 line is ignored)
# see https://askubuntu.com/questions/591937/no-value-for-term-and-no-t-specified
[[ $- == *i* ]] || return

# for nice color indication who you are
export PS1="\[$(tput bold)\]\[\033[38;5;1m\]\u\[$(tput bold)\]\[\033[38;5;15m\]@\[$(tput bold)\]\[\033[38;5;1m\]\h\[$(tput bold)\]\[\033[38;5;15m\]:\[$(tput sgr0)\]\[$(tput sgr0)\]\[\033[38;5;6m\][\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]\[\033[38;5;6m\]\w\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]\[\033[38;5;6m\]]\$\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]"

create in root’s home vim-backup and swap-file-folder
```
mkdir -p ~/.vim/backup ~/.vim/swp
```

create .vimrc with following content:

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
" => Text, tab and indent related
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
" Use spaces instead of tabs
set expandtab

" Be smart when using tabs ;)
set smarttab

" 1 tab == 4 spaces
set shiftwidth=4
set tabstop=4

"Auto indent
set ai

" Smart indent
set si

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
" => Colors and Fonts
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
" Enable syntax highlighting
syntax enable

" set colorscheme for better readability
colorscheme delek

" Set utf8 as standard encoding and en_US as the standard language
set encoding=utf8

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
" => VIM user interface
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
"Always show current position
set ruler

" Ignore case when searching
set ignorecase

" Highlight search results
set hlsearch

" For regular expressions turn magic on
set magic

" Show matching brackets when text indicator is over them
set showmatch

" How many tenths of a second to blink when matching brackets
set mat=2

" No annoying sound on errors
set noerrorbells
set novisualbell
set t_vb=
set tm=500

" Show line numbers
"set number

" Set paste modus always on
set paste

" Jump to last known position
au BufReadPost * if line("'\"") > 1 && line("'\"") <= line("$") | exe "normal! g`\"" | endif

" set backup and swap directory - do not forget to create the folders in the home-folder: mkdir -p ~/.vim/backup ~/.vim/swp
set backup
set backupdir=~/.vim/backup/
set writebackup
set backupcopy=yes
set backupext=.bak
set directory=~/.vim/swp/

" fix for spaces instead of tabs from Fedora 32 on
augroup expandtabFTW
    autocmd!
    autocmd VimEnter * set expandtab
augroup END

generate ssh-keys for later usage, create authorized_keys file and add keys to it in .ssh folder and don’t forget to set permission on authorized_keys to 600
disable selinux by setting SELINUX=disabled in /etc/selinux/config
edit /etc/sshd/sshd_config – enable X11forwarding and disable UseDNS

edit /etc/default/grub and set

GRUB_CMDLINE_LINUX="resume=/dev/mapper/vg_homeserver-lv_swap rd.lvm.lv=vg_homeserver/lv_root rd.lvm.lv=vg_homeserver/lv_swap nomodeset ipv6.disable=1 acpi_enforce_resources=lax"

update grub with:

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

to completely make sure ipv6 is disabled create /etc/sysctl.d/00-ipv6-disable.conf

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

Prepare Network

For that we have to use unfortunately NetworkManagemer. We have to first start and enable openvswitch

systemctl --now enable openvswitch

Then we create one “LAN”-vSwitch, “DMZ”-vSwitch and a “WAN”-vSwitch (or bridges). Each of them connected a physical interface.

LAN-vSwitch

# with this command we remove all config from the main interface (IPs etc. was set by DHCP, if you did a static config at installation resolve all errors popping up first ;) if you want to set this)
nmcli conn modify enp35s0f0 ipv4.method disabled
# we create our vswitch
nmcli conn add type ovs-bridge conn.interface lanbr0
# we create our first port
nmcli conn add type ovs-port conn.interface lanbr0-port0 master lanbr0
# we create an interface "connected" to the port
nmcli conn add type ovs-interface slave-type ovs-port conn.interface lanbr0-int0 master lanbr0-port0 ipv4.method manual ipv4.addresses 192.168.100.240/24 ipv4.gateway 192.168.100.10 ipv4.dns 192.168.100.10
# we create another port
nmcli conn add type ovs-port conn.interface lanbr0-port1 master lanbr0
# we connect the physical interface
nmcli conn add type ethernet conn.interface enp35s0f0 master lanbr0-port1

After rebooting or restarting NetworkManager we should still be able to reach the internet via the old pfSense running on the old lab-hardware

DMZ-vSwitch

This vSwitch is similar but we do not need to set any IPs

nmcli conn add type ovs-bridge conn.interface dmzbr0
nmcli conn add type ovs-port conn.interface dmzbr0-port0 master dmzbr0
nmcli conn add type ovs-interface slave-type ovs-port conn.interface dmzbr0-int0 master dmzbr0-port0 ipv4.method disabled ipv6.method disabled
nmcli conn add type ovs-port conn.interface dmzbr0-port1 master dmzbr0
nmcli conn add type ethernet conn.interface enp35s0f1 master dmzbr0-port1

WAN-vSwitch

And finally the WAN-vSwitch

nmcli conn add type ovs-bridge conn.interface wanbr0
nmcli conn add type ovs-port conn.interface wanbr0-port0 master wanbr0
nmcli conn add type ovs-interface slave-type ovs-port conn.interface wanbr0-int0 master wanbr0-port0 ipv4.method disabled ipv6.method disabled
nmcli conn add type ovs-port conn.interface wanbr0-port1 master wanbr0
nmcli conn add type ethernet conn.interface enp36s0f1 master wanbr0-port1

Disable everywhere ipv6 to be safe and sure

e.g. with:

nmcli conn modify ovs-slave-lanbr0-int0 ipv6.method disabled

NetworkManager Config

create /etc/NetworkManager/conf.d/00-server.conf

[main]
# Do not do automatic (DHCP/SLAAC) configuration on ethernet devices
# with no other matching connections.
no-auto-default=*

create /etc/NetworkManager/conf.d/99-carrier.conf
```
[main]
ignore-carrier=no
```

Reboot and make sure all needed interfaces are listed if issuing command: ip ad

KVM Management

To make sure that VMs get shutdown properly after issuing a reboot or shutdown, we have to edit a few lines in /etc/sysconfig/libvirt-guests

ON_SHUTDOWN=shutdown
PARALLEL_SHUTDOWN=1
SHUTDOWN_TIMEOUT=300

and enable libvirt-guests service

systemctl --now enable libvirt-guests

Create pfSense VM

Download the ISO to the same version used on old lab. Startup virt-manager and create a proper VM for pfSense and set it to start up automatically.

Hints: Choose the correct FreeBSD version pfSense is based on in the creation-wizard. ~~Add the interfaces as e1000 “hardware”.~~ Use virtio-interfaces and disable “Disable hardware checksum offload” in the “System/Advanced/Networking”-options later.

Boot up, install, set the interfaces do initial minimal config and restore the config from the old pfSense. You’ll have to update the interfaces prolly again.

Shutdown/Disconnect old pfSense before really starting the new one. Internet should be reachable from your client machine (not the host where the VM is running on)

Enable VM-Host to reach the Internet again

Because we use those macvtap interfaces for our VM-guests the host cannot reach the set gateway anymore – which is the pfSense.

For being able to reach the outside world again, create a temporary isolated.xml file

<network>
 <name>isolated</name>
 <ip address='192.168.254.1' netmask='255.255.255.0'>
  <dhcp>
   <range start='192.168.254.20' end='192.168.254.254' />
  </dhcp>
 </ip>
</network>

use this file to define via virsh a new network
```
virsh net-define isolated.xml
```
set it to autostart
```
virsh net-autostart isolated
```
start it
```
virsh net-start --network isolated
```
stop pfSense and add this as a new interface
start pfSense add the new interface and configure it with a static IP, e.g. 192.168.254.10
configure that everything from this network may reach the outside – similar to the LAN-firewall rules

remove the gateway from the main-interface

nmcli conn modify ovs-slave-lanbr0-int0 ipv4.gateway ""

we now can set a temporary new default gateway with
```
ip route add default via 192.168.254.10 dev virbr1
```

Make new default gateway permanent on host

as we are missusing the “isolated” virsh network for getting outside, we cannot set the default gateway nicely via virsh afaik or nmcli. After a reboot the default gateway would be missing again. So we use systemd to set the route after everything is started. To make sure this is really the last command, we have to do following:

create a /etc/systemd/system/custom.target file

[Unit]
Description=Custom Target
Requires=multi-user.target
After=multi-user.target
AllowIsolate=yes

create a /etc/systemd/system/setup-default-gateway.service file

[Unit]
Description=Setup Default Gateway
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c "ip route add default via 192.168.254.10 dev virbr1"
RemainAfterExit=true
ExecStop=/bin/bash -c "ip route del default"
StandardOutput=journal

[Install]
WantedBy=custom.target

create a folder /etc/systemd/system/custom.target.wants

create a symbolic link

ln -s /etc/systemd/system/setup-default-gateway.service /etc/systemd/system/custom.target.wants/setup-default-gateway.service

create a backup of /etc/systemd/system/default.target
reload systemd
```
systemctl daemon-reload
```
set it to default.target
```
systemctl set-default custom.target
```

Reboot and test if everything is OK

My VM New Homelab