19 Jan

Freebsd 11.0 – LanCache

During one of the lan events I was hosting for my students I noticed the total bandwidth for 3 days, it was very high. In this case it was 2,15TB in total of data transferred in just a couple of days. No one complaint because the event was hosted on a 500/500Mbit fiber connection, but if I were to extend the event with more than 35 users problem would arise.

First though was a squid proxy, but since a lot of the data was from steam or similar game services I started to look into something called LAN-Caching. Which should cache as much as possible. I ended up with the configuration bellow, I will use that in the future to reduce bandwidth usage on the wan interface. And this configuration will be used on 2 identical high-perfomance servers connected to a core Cisco switch.

There are some improvements that I still have todo with this setup,I’m still doing try runs with some specific users to learn which service fails and what I fail to cache. This Isn’t a final build and must always be optimised and changed before an event.

Stats from the first event:

Network drawing – Practical example

Requirments
Complete hdd/ssd setup & mount it as /data. 
Complete the network structure Gateway + switches(read about lagg).
Fresh install of unbound or selected it during freebsd 11.0 install.
First install nginx – Server 1 & 2
# Select basic settings + add rewrite module + ssl + slice
cd /usr/ports/www/nginx
make config-recursive
make install clean
echo 'nginx_enable="YES"' >> /etc/rc.conf
/boot/loader.conf – Server 1 & 2
# Load carp module
carp_load="YES"
# Buffer incoming connections until certain http request arrives 
accf_http_load="YES"
# Wait for data accept filter
accf_data_load="YES"
# Load lagg module
if_lagg_load="YES"
/etc/sysctl.conf – Server 1 & 2
# Carp settings
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=1
# Allow ip forwarding
net.inet.ip.forwarding=1
/etc/rc.conf – Server 1
# LAGG & Carp Server 1
ifconfig_bce0="up"
ifconfig_bce1="up"
defaultrouter="10.0.2.1"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport bce0 laggport bce1 10.0.2.2/24 up"

ifconfig_lagg0_aliases="\
        inet vhid 1 advskew 100 pass paswd alias 10.0.2.4/32 \
        inet vhid 2 advskew 100 pass paswd alias 10.0.2.5/32 \
        inet vhid 3 advskew 200 pass paswd alias 10.0.2.6/32 \
        inet vhid 4 advskew 100 pass paswd alias 10.0.2.7/32 \
        inet vhid 5 advskew 200 pass paswd alias 10.0.2.8/32 \
        inet vhid 6 advskew 100 pass paswd alias 10.0.2.9/32 \
        inet vhid 7 advskew 200 pass paswd alias 10.0.2.10/32 \
        inet vhid 8 advskew 100 pass paswd alias 10.0.2.11/32 \
        inet vhid 9 advskew 200 pass paswd alias 10.0.2.12/32 \
        inet vhid 10 advskew 100 pass paswd alias 10.0.2.13/32 \
        inet vhid 11 advskew 200 pass paswd alias 10.0.2.14/32 \
        inet vhid 12 advskew 100 pass paswd alias 10.0.2.15/32"
/etc/rc.conf – Server 2
# LAGG & Carp Server 2
ifconfig_bce0="up"
ifconfig_bce1="up"
defaultrouter="10.0.2.1"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport bce0 laggport bce1 10.0.2.3/24 up"

ifconfig_lagg0_aliases="\
        inet vhid 1 advskew 200 pass paswd alias 10.0.2.4/32 \
        inet vhid 2 advskew 200 pass paswd alias 10.0.2.5/32 \
        inet vhid 3 advskew 100 pass paswd alias 10.0.2.6/32 \
        inet vhid 4 advskew 200 pass paswd alias 10.0.2.7/32 \
        inet vhid 5 advskew 100 pass paswd alias 10.0.2.8/32 \
        inet vhid 6 advskew 200 pass paswd alias 10.0.2.9/32 \
        inet vhid 7 advskew 100 pass paswd alias 10.0.2.10/32 \
        inet vhid 8 advskew 200 pass paswd alias 10.0.2.11/32 \
        inet vhid 9 advskew 100 pass paswd alias 10.0.2.12/32 \
        inet vhid 10 advskew 200 pass paswd alias 10.0.2.13/32 \
        inet vhid 11 advskew 100 pass paswd alias 10.0.2.14/32 \
        inet vhid 12 advskew 200 pass paswd alias 10.0.2.15/32"
/etc/hosts – Server 1 & 2
# Link some host names to specific ip's, can be used in unbound and nginx
10.0.2.5        lancache-steam
10.0.2.6        lancache-riot
10.0.2.7        lancache-blizzard
10.0.2.8        lancache-hirez
10.0.2.9        lancache-origin
10.0.2.10       lancache-sony
10.0.2.11       lancache-arenanetworks
10.0.2.12       lancache-ubisoft
10.0.2.13       lancache-gog
10.0.2.14       lancache-turbine
10.0.2.15       lancache-microsoft
/etc/unbound/unbound.conf – Server 1 & 2
# Basic settings
server:
        interface:10.0.2.2
        interface:0.0.0.0
        interface:10.0.2.4
        access-control: 0.0.0.0/0 allow
        private-address: 10.0.2.0/24
        ip-transparent: yes
        do-ip4: yes
        do-udp: yes
        do-tcp:yes
        do-daemonize:yes
        username: unbound
        directory: /var/unbound
        chroot: /var/unbound
        pidfile: /var/run/local_unbound.pid
        auto-trust-anchor-file: /var/unbound/root.key

include: /var/unbound/lancaching.conf
include: /var/unbound/forward.conf
include: /var/unbound/lan-zones.conf
include: /var/unbound/control.conf
include: /var/unbound/conf.d/*.conf

Fetching lancaching.conf – Server 1 & 2
# Enter the unbound folder, fetch the file: lancaching.conf
cd /var/unbound/
fetch --no-verify-peer http://lyxi.ga/wp-content/uploads/2017/lancaching.conf
Nginx cache folders + logs – Server 1 & 2
# Create folders for logs and cache data
mkdir /data/ | mkdir /data/www/
mkdir /data/www/logs/ | mkdir /data/www/cache/
mkdir /data/www/cache/tmp | mkdir /data/www/cache/other | mkdir /data/www/cache/installs

# Change owner of the folder and set full permission on /data
chown -R www:www /data | chmod -R 777 /data

# Download the nginx configs for lan-cache, remove old nginx.conf & unpack.
rm /usr/local/etc/nginx/nginx.conf
cd /tmp/
# This contains a modified version of junkhacker's lancache,to work on freebsd.
fetch --no-verify-peer  http://lyxi.ga/wp-content/uploads/2017/lancachemaster.tar.gz
tar -zxf lancachemaster.tar.gz
cp -R /tmp/lancachemaster/* /usr/local/etc/nginx/
rm -R /tmp/lancachemaster | rm lancachemaster.tar.gz

# Original source of configs bellow for latest update: 
https://github.com/junkhacker/lancache

# Rebooting nginx 
/usr/local/etc/rc.d/nginx restart
Try it out
# Good to use command to monitor current connection speed 
systat -ifstat

# Once everything is setup and default dns is set to 10.0.2.4 you should be able to
# launch steam, do a fresh download, remove the game from library and download again.
# Result should be something similar as the photos bellow.

# This is by no mean optimal, modifications need to be done a couple of days before the event.

Photo before files cached:

Photo after files were cached:

Sources

https://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/
https://github.com/ForayJones/lancache
https://github.com/junkhacker/lancache
https://github.com/bntjah/lancache
https://blog.yolocation.pro/index.php/2016/02/03/how-to-install-lancache-on-debian/

Leave a Reply

Your email address will not be published. Required fields are marked *