This HowTo is mainly meant as documentation if I have to resetup my lab 🙂

Hardware

  • Ryzen 7 4750G
  • 64GB RAM
  • Network Interface card with 4 ports
  • 1TB NVME M2 disk

Prerequisites

  • Internet Connection (in this case via my pfSense VM running on old lab)
  • Backup of pfSense config

Setup

  • After doing a minimal Fedora 32 Server installation change to root-user and change hostname to something pretty to show in console
    hostnamectl set-hostname homeserver
  • Install some stuff needed later and do an update after that
    dnf install vim-enhanced virt-manager libvirt-client NetworkManager-ovs xorg-x11-server-Xorg xorg-x11-xauth
  • Add to .bashrc
    alias vi='vim'
    
    # to resolv "tput: No value for $TERM and no -T specified" messages if you use scp for example (checks if session is interactive, if not the PS1 line is ignored)
    # see https://askubuntu.com/questions/591937/no-value-for-term-and-no-t-specified
    [[ $- == *i* ]] || return
    
    # for nice color indication who you are
    export PS1="\[$(tput bold)\]\[\033[38;5;1m\]\u\[$(tput bold)\]\[\033[38;5;15m\]@\[$(tput bold)\]\[\033[38;5;1m\]\h\[$(tput bold)\]\[\033[38;5;15m\]:\[$(tput sgr0)\]\[$(tput sgr0)\]\[\033[38;5;6m\][\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]\[\033[38;5;6m\]\w\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]\[\033[38;5;6m\]]\$\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]"
  • create in root’s home vim-backup and swap-file-folder
    mkdir -p ~/.vim/backup ~/.vim/swp
  • create .vimrc with following content:
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
    " => Text, tab and indent related
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
    " Use spaces instead of tabs
    set expandtab
    
    " Be smart when using tabs ;)
    set smarttab
    
    " 1 tab == 4 spaces
    set shiftwidth=4
    set tabstop=4
    
    "Auto indent
    set ai
    
    " Smart indent
    set si
    
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
    " => Colors and Fonts
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
    " Enable syntax highlighting
    syntax enable
    
    " set colorscheme for better readability
    colorscheme delek
    
    " Set utf8 as standard encoding and en_US as the standard language
    set encoding=utf8
    
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
    " => VIM user interface
    """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
    "Always show current position
    set ruler
    
    " Ignore case when searching
    set ignorecase
    
    " Highlight search results
    set hlsearch
    
    " For regular expressions turn magic on
    set magic
    
    " Show matching brackets when text indicator is over them
    set showmatch
    
    " How many tenths of a second to blink when matching brackets
    set mat=2
    
    " No annoying sound on errors
    set noerrorbells
    set novisualbell
    set t_vb=
    set tm=500
    
    " Show line numbers
    "set number
    
    " Set paste modus always on
    set paste
    
    " Jump to last known position
    au BufReadPost * if line("'\"") > 1 && line("'\"") <= line("$") | exe "normal! g`\"" | endif
    
    " set backup and swap directory - do not forget to create the folders in the home-folder: mkdir -p ~/.vim/backup ~/.vim/swp
    set backup
    set backupdir=~/.vim/backup/
    set writebackup
    set backupcopy=yes
    set backupext=.bak
    set directory=~/.vim/swp/
    
    " fix for spaces instead of tabs from Fedora 32 on
    augroup expandtabFTW
        autocmd!
        autocmd VimEnter * set expandtab
    augroup END
  • generate ssh-keys for later usage, create authorized_keys file and add keys to it in .ssh folder and don’t forget to set permission on authorized_keys to 600
  • disable selinux by setting SELINUX=disabled in /etc/selinux/config
  • edit /etc/sshd/sshd_config – enable X11forwarding and disable UseDNS
  • edit /etc/default/grub and set
    GRUB_CMDLINE_LINUX="resume=/dev/mapper/vg_homeserver-lv_swap rd.lvm.lv=vg_homeserver/lv_root rd.lvm.lv=vg_homeserver/lv_swap nomodeset ipv6.disable=1 acpi_enforce_resources=lax"
  • update grub with:
    grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
  • to completely make sure ipv6 is disabled create /etc/sysctl.d/00-ipv6-disable.conf
    net.ipv6.conf.all.disable_ipv6=1
    net.ipv6.conf.default.disable_ipv6=1
    net.ipv6.conf.lo.disable_ipv6=1

Prepare Network

For that we have to use unfortunately NetworkManagemer. We have to first start and enable openvswitch

systemctl --now enable openvswitch

Then we create  one „LAN“-vSwitch, „DMZ“-vSwitch and a „WAN“-vSwitch (or bridges). Each of them connected a physical interface.

LAN-vSwitch

# with this command we remove all config from the main interface (IPs etc. was set by DHCP, if you did a static config at installation resolve all errors popping up first ;) if you want to set this)
nmcli conn modify enp35s0f0 ipv4.method disabled
# we create our vswitch
nmcli conn add type ovs-bridge conn.interface lanbr0
# we create our first port
nmcli conn add type ovs-port conn.interface lanbr0-port0 master lanbr0
# we create an interface "connected" to the port
nmcli conn add type ovs-interface slave-type ovs-port conn.interface lanbr0-int0 master lanbr0-port0 ipv4.method manual ipv4.addresses 192.168.100.240/24 ipv4.gateway 192.168.100.10 ipv4.dns 192.168.100.10
# we create another port
nmcli conn add type ovs-port conn.interface lanbr0-port1 master lanbr0
# we connect the physical interface
nmcli conn add type ethernet conn.interface enp35s0f0 master lanbr0-port1

After rebooting or restarting NetworkManager we should still be able to reach the internet via the old pfSense running on the old lab-hardware

DMZ-vSwitch

This vSwitch is similar but we do not need to set any IPs

nmcli conn add type ovs-bridge conn.interface dmzbr0
nmcli conn add type ovs-port conn.interface dmzbr0-port0 master dmzbr0
nmcli conn add type ovs-interface slave-type ovs-port conn.interface dmzbr0-int0 master dmzbr0-port0 ipv4.method disabled ipv6.method disabled
nmcli conn add type ovs-port conn.interface dmzbr0-port1 master dmzbr0
nmcli conn add type ethernet conn.interface enp35s0f1 master dmzbr0-port1

WAN-vSwitch

And finally the WAN-vSwitch

nmcli conn add type ovs-bridge conn.interface wanbr0
nmcli conn add type ovs-port conn.interface wanbr0-port0 master wanbr0
nmcli conn add type ovs-interface slave-type ovs-port conn.interface wanbr0-int0 master wanbr0-port0 ipv4.method disabled ipv6.method disabled
nmcli conn add type ovs-port conn.interface wanbr0-port1 master wanbr0
nmcli conn add type ethernet conn.interface enp36s0f1 master wanbr0-port1

Disable everywhere ipv6 to be safe and sure

e.g. with:

nmcli conn modify ovs-slave-lanbr0-int0 ipv6.method disabled

NetworkManager Config

  • create /etc/NetworkManager/conf.d/00-server.conf
    [main]
    # Do not do automatic (DHCP/SLAAC) configuration on ethernet devices
    # with no other matching connections.
    no-auto-default=*
  • create /etc/NetworkManager/conf.d/99-carrier.conf
    [main]
    ignore-carrier=no

Reboot and make sure all needed interfaces are listed if issuing command: ip ad

KVM Management

To make sure that VMs get shutdown properly after issuing a reboot or shutdown, we have to edit a few lines in /etc/sysconfig/libvirt-guests

ON_SHUTDOWN=shutdown
PARALLEL_SHUTDOWN=1
SHUTDOWN_TIMEOUT=300

and enable libvirt-guests service

systemctl --now enable libvirt-guests

Create pfSense VM

Download the ISO to the same version used on old lab. Startup virt-manager and create a proper VM for pfSense and set it to start up automatically.

Hints: Choose the correct FreeBSD version pfSense is based on in the creation-wizard. Add the interfaces as e1000 „hardware“. Use virtio-interfaces and disable „Disable hardware checksum offload“ in the „System/Advanced/Networking“-options later.

Boot up, install, set the interfaces do initial minimal config and restore the config from the old pfSense. You’ll have to update the interfaces prolly again.

Shutdown/Disconnect old pfSense before really starting the new one. Internet should be reachable from your client machine (not the host where the VM is running on)

Enable VM-Host to reach the Internet again

Because we use those macvtap interfaces for our VM-guests the host cannot reach the set gateway anymore – which is the pfSense.

  • For being able to reach the outside world again, create a temporary isolated.xml file
    <network>
     <name>isolated</name>
     <ip address='192.168.254.1' netmask='255.255.255.0'>
      <dhcp>
       <range start='192.168.254.20' end='192.168.254.254' />
      </dhcp>
     </ip>
    </network>
  • use this file to define via virsh a new network
    virsh net-define isolated.xml
  • set it to autostart
    virsh net-autostart isolated
  • start it
    virsh net-start --network isolated
  • stop pfSense and add this as a new interface
  • start pfSense add the new interface and configure it with a static IP, e.g. 192.168.254.10
  • configure that everything from this network may reach the outside – similar to the LAN-firewall rules
  • remove the gateway from the main-interface
    nmcli conn modify ovs-slave-lanbr0-int0 ipv4.gateway ""
  • we now can set a temporary new default gateway with
    ip route add default via 192.168.254.10 dev virbr1

Make new default gateway permanent on host

as we are missusing the „isolated“ virsh network for getting outside, we cannot set the default gateway nicely via virsh afaik or nmcli. After a reboot the default gateway would be missing again. So we use systemd to set the route after everything is started. To make sure this is really the last command, we have to do following:

  • create a /etc/systemd/system/custom.target file
    [Unit]
    Description=Custom Target
    Requires=multi-user.target
    After=multi-user.target
    AllowIsolate=yes
  • create a /etc/systemd/system/setup-default-gateway.service file
    [Unit]
    Description=Setup Default Gateway
    After=multi-user.target
    
    [Service]
    Type=oneshot
    ExecStart=/bin/bash -c "ip route add default via 192.168.254.10 dev virbr1"
    RemainAfterExit=true
    ExecStop=/bin/bash -c "ip route del default"
    StandardOutput=journal
    
    [Install]
    WantedBy=custom.target
  • create a folder /etc/systemd/system/custom.target.wants
  • create a symbolic link
    ln -s /etc/systemd/system/setup-default-gateway.service /etc/systemd/system/custom.target.wants/setup-default-gateway.service
  • create a backup of /etc/systemd/system/default.target
  • reload systemd
    systemctl daemon-reload
  • set it to default.target
    systemctl set-default custom.target

Reboot and test if everything is OK

Links

Zuletzt bearbeitet: August 5, 2021

Autor

Kommentare

Kommentar verfassen

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden.