This HowTo is mainly meant as documentation if I have to resetup my lab 🙂
Hardware
- Ryzen 7 4750G
- 64GB RAM
- Network Interface card with 4 ports
- 1TB NVME M2 disk
Prerequisites
- Internet Connection (in this case via my pfSense VM running on old lab)
- Backup of pfSense config
Setup
- After doing a minimal Fedora 32 Server installation change to root-user and change hostname to something pretty to show in console
hostnamectl set-hostname homeserver
- Install some stuff needed later and do an update after that
dnf install vim-enhanced virt-manager libvirt-client NetworkManager-ovs xorg-x11-server-Xorg xorg-x11-xauth
- Add to .bashrc
alias vi='vim' # to resolv "tput: No value for $TERM and no -T specified" messages if you use scp for example (checks if session is interactive, if not the PS1 line is ignored) # see https://askubuntu.com/questions/591937/no-value-for-term-and-no-t-specified [[ $- == *i* ]] || return # for nice color indication who you are export PS1="\[$(tput bold)\]\[\033[38;5;1m\]\u\[$(tput bold)\]\[\033[38;5;15m\]@\[$(tput bold)\]\[\033[38;5;1m\]\h\[$(tput bold)\]\[\033[38;5;15m\]:\[$(tput sgr0)\]\[$(tput sgr0)\]\[\033[38;5;6m\][\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]\[\033[38;5;6m\]\w\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]\[\033[38;5;6m\]]\$\[$(tput sgr0)\]\[\033[38;5;15m\] \[$(tput sgr0)\]"
- create in root’s home vim-backup and swap-file-folder
mkdir -p ~/.vim/backup ~/.vim/swp
- create
.vimrc
with following content:""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" " => Text, tab and indent related """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" " Use spaces instead of tabs set expandtab " Be smart when using tabs ;) set smarttab " 1 tab == 4 spaces set shiftwidth=4 set tabstop=4 "Auto indent set ai " Smart indent set si """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" " => Colors and Fonts """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" " Enable syntax highlighting syntax enable " set colorscheme for better readability colorscheme delek " Set utf8 as standard encoding and en_US as the standard language set encoding=utf8 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" " => VIM user interface """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" "Always show current position set ruler " Ignore case when searching set ignorecase " Highlight search results set hlsearch " For regular expressions turn magic on set magic " Show matching brackets when text indicator is over them set showmatch " How many tenths of a second to blink when matching brackets set mat=2 " No annoying sound on errors set noerrorbells set novisualbell set t_vb= set tm=500 " Show line numbers "set number " Set paste modus always on set paste " Jump to last known position au BufReadPost * if line("'\"") > 1 && line("'\"") <= line("$") | exe "normal! g`\"" | endif " set backup and swap directory - do not forget to create the folders in the home-folder: mkdir -p ~/.vim/backup ~/.vim/swp set backup set backupdir=~/.vim/backup/ set writebackup set backupcopy=yes set backupext=.bak set directory=~/.vim/swp/ " fix for spaces instead of tabs from Fedora 32 on augroup expandtabFTW autocmd! autocmd VimEnter * set expandtab augroup END
- generate ssh-keys for later usage, create
authorized_keys
file and add keys to it in.ssh
folder and don’t forget to set permission onauthorized_keys
to600
- disable selinux by setting
SELINUX=disabled
in/etc/selinux/config
- edit
/etc/sshd/sshd_config
– enableX11forwarding
and disableUseDNS
- edit
/etc/default/grub
and setGRUB_CMDLINE_LINUX="resume=/dev/mapper/vg_homeserver-lv_swap rd.lvm.lv=vg_homeserver/lv_root rd.lvm.lv=vg_homeserver/lv_swap nomodeset ipv6.disable=1 acpi_enforce_resources=lax"
- update grub with:
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
- to completely make sure ipv6 is disabled create
/etc/sysctl.d/00-ipv6-disable.conf
net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1
Prepare Network
For that we have to use unfortunately NetworkManagemer. We have to first start and enable openvswitch
systemctl --now enable openvswitch
Then we create one „LAN“-vSwitch, „DMZ“-vSwitch and a „WAN“-vSwitch (or bridges). Each of them connected a physical interface.
LAN-vSwitch
# with this command we remove all config from the main interface (IPs etc. was set by DHCP, if you did a static config at installation resolve all errors popping up first ;) if you want to set this) nmcli conn modify enp35s0f0 ipv4.method disabled # we create our vswitch nmcli conn add type ovs-bridge conn.interface lanbr0 # we create our first port nmcli conn add type ovs-port conn.interface lanbr0-port0 master lanbr0 # we create an interface "connected" to the port nmcli conn add type ovs-interface slave-type ovs-port conn.interface lanbr0-int0 master lanbr0-port0 ipv4.method manual ipv4.addresses 192.168.100.240/24 ipv4.gateway 192.168.100.10 ipv4.dns 192.168.100.10 # we create another port nmcli conn add type ovs-port conn.interface lanbr0-port1 master lanbr0 # we connect the physical interface nmcli conn add type ethernet conn.interface enp35s0f0 master lanbr0-port1
After rebooting or restarting NetworkManager we should still be able to reach the internet via the old pfSense running on the old lab-hardware
DMZ-vSwitch
This vSwitch is similar but we do not need to set any IPs
nmcli conn add type ovs-bridge conn.interface dmzbr0 nmcli conn add type ovs-port conn.interface dmzbr0-port0 master dmzbr0 nmcli conn add type ovs-interface slave-type ovs-port conn.interface dmzbr0-int0 master dmzbr0-port0 ipv4.method disabled ipv6.method disabled nmcli conn add type ovs-port conn.interface dmzbr0-port1 master dmzbr0 nmcli conn add type ethernet conn.interface enp35s0f1 master dmzbr0-port1
WAN-vSwitch
And finally the WAN-vSwitch
nmcli conn add type ovs-bridge conn.interface wanbr0 nmcli conn add type ovs-port conn.interface wanbr0-port0 master wanbr0 nmcli conn add type ovs-interface slave-type ovs-port conn.interface wanbr0-int0 master wanbr0-port0 ipv4.method disabled ipv6.method disabled nmcli conn add type ovs-port conn.interface wanbr0-port1 master wanbr0 nmcli conn add type ethernet conn.interface enp36s0f1 master wanbr0-port1
Disable everywhere ipv6 to be safe and sure
e.g. with:
nmcli conn modify ovs-slave-lanbr0-int0 ipv6.method disabled
NetworkManager Config
- create
/etc/NetworkManager/conf.d/00-server.conf
[main] # Do not do automatic (DHCP/SLAAC) configuration on ethernet devices # with no other matching connections. no-auto-default=*
- create
/etc/NetworkManager/conf.d/99-carrier.conf
[main] ignore-carrier=no
Reboot and make sure all needed interfaces are listed if issuing command: ip ad
KVM Management
To make sure that VMs get shutdown properly after issuing a reboot or shutdown, we have to edit a few lines in /etc/sysconfig/libvirt-guests
ON_SHUTDOWN=shutdown PARALLEL_SHUTDOWN=1 SHUTDOWN_TIMEOUT=300
and enable libvirt-guests service
systemctl --now enable libvirt-guests
Create pfSense VM
Download the ISO to the same version used on old lab. Startup virt-manager and create a proper VM for pfSense and set it to start up automatically.
Hints: Choose the correct FreeBSD version pfSense is based on in the creation-wizard.
Add the interfaces as e1000 „hardware“.Use virtio-interfaces and disable „Disable hardware checksum offload“ in the „System/Advanced/Networking“-options later.
Boot up, install, set the interfaces do initial minimal config and restore the config from the old pfSense. You’ll have to update the interfaces prolly again.
Shutdown/Disconnect old pfSense before really starting the new one. Internet should be reachable from your client machine (not the host where the VM is running on)
Enable VM-Host to reach the Internet again
Because we use those macvtap
interfaces for our VM-guests the host cannot reach the set gateway anymore – which is the pfSense.
- For being able to reach the outside world again, create a temporary
isolated.xml
file<network> <name>isolated</name> <ip address='192.168.254.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.254.20' end='192.168.254.254' /> </dhcp> </ip> </network>
- use this file to define via
virsh
a new networkvirsh net-define isolated.xml
- set it to autostart
virsh net-autostart isolated
- start it
virsh net-start --network isolated
- stop pfSense and add this as a new interface
- start pfSense add the new interface and configure it with a static IP, e.g. 192.168.254.10
- configure that everything from this network may reach the outside – similar to the LAN-firewall rules
- remove the gateway from the main-interface
nmcli conn modify ovs-slave-lanbr0-int0 ipv4.gateway ""
- we now can set a temporary new default gateway with
ip route add default via 192.168.254.10 dev virbr1
Make new default gateway permanent on host
as we are missusing the „isolated“ virsh network for getting outside, we cannot set the default gateway nicely via virsh
afaik or nmcli
. After a reboot the default gateway would be missing again. So we use systemd
to set the route after everything is started. To make sure this is really the last command, we have to do following:
- create a
/etc/systemd/system/custom.target
file[Unit] Description=Custom Target Requires=multi-user.target After=multi-user.target AllowIsolate=yes
- create a
/etc/systemd/system/setup-default-gateway.service
file[Unit] Description=Setup Default Gateway After=multi-user.target [Service] Type=oneshot ExecStart=/bin/bash -c "ip route add default via 192.168.254.10 dev virbr1" RemainAfterExit=true ExecStop=/bin/bash -c "ip route del default" StandardOutput=journal [Install] WantedBy=custom.target
- create a folder
/etc/systemd/system/custom.target.wants
- create a symbolic link
ln -s /etc/systemd/system/setup-default-gateway.service /etc/systemd/system/custom.target.wants/setup-default-gateway.service
- create a backup of
/etc/systemd/system/default.target
- reload systemd
systemctl daemon-reload
- set it to
default.target
systemctl set-default custom.target
Reboot and test if everything is OK
Links
- https://superuser.com/questions/544399/how-do-you-make-a-systemd-service-as-the-last-service-on-boot
- https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_host_configuration_and_guest_installation_guide/app_macvtap
- https://developer.gnome.org/NetworkManager/stable/nm-openvswitch.html
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_administration_guide/sub-sect-shutting_down_rebooting_and_force_shutdown_of_a_guest_virtual_machine-manipulating_the_libvirt_guests_configuration_settings
Kommentare