19 Jan

Freebsd 11.0 – LanCache

During one of the lan events I was hosting for my students I noticed the total bandwidth for 3 days, it was very high. In this case it was 2,15TB in total of data transferred in just a couple of days. No one complaint because the event was hosted on a 500/500Mbit fiber connection, but if I were to extend the event with more than 35 users problem would arise.

First though was a squid proxy, but since a lot of the data was from steam or similar game services I started to look into something called LAN-Caching. Which should cache as much as possible. I ended up with the configuration bellow, I will use that in the future to reduce bandwidth usage on the wan interface. And this configuration will be used on 2 identical high-perfomance servers connected to a core Cisco switch.

There are some improvements that I still have todo with this setup,I’m still doing try runs with some specific users to learn which service fails and what I fail to cache. This Isn’t a final build and must always be optimised and changed before an event.

Stats from the first event:

Network drawing – Practical example

Requirments
Complete hdd/ssd setup & mount it as /data. 
Complete the network structure Gateway + switches(read about lagg).
Fresh install of unbound or selected it during freebsd 11.0 install.
First install nginx – Server 1 & 2
# Select basic settings + add rewrite module + ssl + slice
cd /usr/ports/www/nginx
make config-recursive
make install clean
echo 'nginx_enable="YES"' >> /etc/rc.conf
/boot/loader.conf – Server 1 & 2
# Load carp module
carp_load="YES"
# Buffer incoming connections until certain http request arrives 
accf_http_load="YES"
# Wait for data accept filter
accf_data_load="YES"
# Load lagg module
if_lagg_load="YES"
/etc/sysctl.conf – Server 1 & 2
# Carp settings
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=1
# Allow ip forwarding
net.inet.ip.forwarding=1
/etc/rc.conf – Server 1
# LAGG & Carp Server 1
ifconfig_bce0="up"
ifconfig_bce1="up"
defaultrouter="10.0.2.1"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport bce0 laggport bce1 10.0.2.2/24 up"

ifconfig_lagg0_aliases="\
        inet vhid 1 advskew 100 pass paswd alias 10.0.2.4/32 \
        inet vhid 2 advskew 100 pass paswd alias 10.0.2.5/32 \
        inet vhid 3 advskew 200 pass paswd alias 10.0.2.6/32 \
        inet vhid 4 advskew 100 pass paswd alias 10.0.2.7/32 \
        inet vhid 5 advskew 200 pass paswd alias 10.0.2.8/32 \
        inet vhid 6 advskew 100 pass paswd alias 10.0.2.9/32 \
        inet vhid 7 advskew 200 pass paswd alias 10.0.2.10/32 \
        inet vhid 8 advskew 100 pass paswd alias 10.0.2.11/32 \
        inet vhid 9 advskew 200 pass paswd alias 10.0.2.12/32 \
        inet vhid 10 advskew 100 pass paswd alias 10.0.2.13/32 \
        inet vhid 11 advskew 200 pass paswd alias 10.0.2.14/32 \
        inet vhid 12 advskew 100 pass paswd alias 10.0.2.15/32"
/etc/rc.conf – Server 2
# LAGG & Carp Server 2
ifconfig_bce0="up"
ifconfig_bce1="up"
defaultrouter="10.0.2.1"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport bce0 laggport bce1 10.0.2.3/24 up"

ifconfig_lagg0_aliases="\
        inet vhid 1 advskew 200 pass paswd alias 10.0.2.4/32 \
        inet vhid 2 advskew 200 pass paswd alias 10.0.2.5/32 \
        inet vhid 3 advskew 100 pass paswd alias 10.0.2.6/32 \
        inet vhid 4 advskew 200 pass paswd alias 10.0.2.7/32 \
        inet vhid 5 advskew 100 pass paswd alias 10.0.2.8/32 \
        inet vhid 6 advskew 200 pass paswd alias 10.0.2.9/32 \
        inet vhid 7 advskew 100 pass paswd alias 10.0.2.10/32 \
        inet vhid 8 advskew 200 pass paswd alias 10.0.2.11/32 \
        inet vhid 9 advskew 100 pass paswd alias 10.0.2.12/32 \
        inet vhid 10 advskew 200 pass paswd alias 10.0.2.13/32 \
        inet vhid 11 advskew 100 pass paswd alias 10.0.2.14/32 \
        inet vhid 12 advskew 200 pass paswd alias 10.0.2.15/32"
/etc/hosts – Server 1 & 2
# Link some host names to specific ip's, can be used in unbound and nginx
10.0.2.5        lancache-steam
10.0.2.6        lancache-riot
10.0.2.7        lancache-blizzard
10.0.2.8        lancache-hirez
10.0.2.9        lancache-origin
10.0.2.10       lancache-sony
10.0.2.11       lancache-arenanetworks
10.0.2.12       lancache-ubisoft
10.0.2.13       lancache-gog
10.0.2.14       lancache-turbine
10.0.2.15       lancache-microsoft
/etc/unbound/unbound.conf – Server 1 & 2
# Basic settings
server:
        interface:10.0.2.2
        interface:0.0.0.0
        interface:10.0.2.4
        access-control: 0.0.0.0/0 allow
        private-address: 10.0.2.0/24
        ip-transparent: yes
        do-ip4: yes
        do-udp: yes
        do-tcp:yes
        do-daemonize:yes
        username: unbound
        directory: /var/unbound
        chroot: /var/unbound
        pidfile: /var/run/local_unbound.pid
        auto-trust-anchor-file: /var/unbound/root.key

include: /var/unbound/lancaching.conf
include: /var/unbound/forward.conf
include: /var/unbound/lan-zones.conf
include: /var/unbound/control.conf
include: /var/unbound/conf.d/*.conf

Fetching lancaching.conf – Server 1 & 2
# Enter the unbound folder, fetch the file: lancaching.conf
cd /var/unbound/
fetch --no-verify-peer http://lyxi.ga/wp-content/uploads/2017/lancaching.conf
Nginx cache folders + logs – Server 1 & 2
# Create folders for logs and cache data
mkdir /data/ | mkdir /data/www/
mkdir /data/www/logs/ | mkdir /data/www/cache/
mkdir /data/www/cache/tmp | mkdir /data/www/cache/other | mkdir /data/www/cache/installs

# Change owner of the folder and set full permission on /data
chown -R www:www /data | chmod -R 777 /data

# Download the nginx configs for lan-cache, remove old nginx.conf & unpack.
rm /usr/local/etc/nginx/nginx.conf
cd /tmp/
# This contains a modified version of junkhacker's lancache,to work on freebsd.
fetch --no-verify-peer  http://lyxi.ga/wp-content/uploads/2017/lancachemaster.tar.gz
tar -zxf lancachemaster.tar.gz
cp -R /tmp/lancachemaster/* /usr/local/etc/nginx/
rm -R /tmp/lancachemaster | rm lancachemaster.tar.gz

# Original source of configs bellow for latest update: 
https://github.com/junkhacker/lancache

# Rebooting nginx 
/usr/local/etc/rc.d/nginx restart
Try it out
# Good to use command to monitor current connection speed 
systat -ifstat

# Once everything is setup and default dns is set to 10.0.2.4 you should be able to
# launch steam, do a fresh download, remove the game from library and download again.
# Result should be something similar as the photos bellow.

# This is by no mean optimal, modifications need to be done a couple of days before the event.

Photo before files cached:

Photo after files were cached:

Sources

https://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/
https://github.com/ForayJones/lancache
https://github.com/junkhacker/lancache
https://github.com/bntjah/lancache
https://blog.yolocation.pro/index.php/2016/02/03/how-to-install-lancache-on-debian/

Junkhacker says:

i was checking out your config, and i thought i should note that the steam servers are region specific. you probably need different servers than i had. there was once a regex capable patch for dnsmasq that would allow the capture of all of the valve.steampipe.steamconent.com addresses, but it’s sadly out of date.
if you do any additional work on this, i’d appreciate you sharing the contribution on github.

Peter Bergman says:

Yapp the steam servers are region specific(same with blizzard), for the test setup i used US servers and configured steam the same way, usually i just setup wireshark and sniff the traffic before an event just to find out what servers i’m hitting. I would gladly contribute on github once i get some more free time to play with it. And i must thank you for the great work you did with the lancache, it was a good read and it helped me a lot while doing this test.

junkhacker says:

don’t thank me, thank the folks at multiplay and those who created/maintained it before. i can take very little credit for it.

Peter Bergman says:

Still the part on github helped me a bit, and just that is worth saying thanks for 🙂 and the folks at multiplay i would also say thanks to but i’m unsure if they were to read this. For me, every small or big contributor is worth thanking, also it’s the reason why i always link the sources i use, no matter how much i use of it.

Leave a Reply

Your email address will not be published. Required fields are marked *