TechOPS

All things related to FNF Technical Operations. This covers all three FNF locations (MCI,AUS,DFW).

Kansas City Point of Presence - Data center documentation

The purpose of this section is to provide documentation of the FNF enterprise infrastructure deployed in Kansas City. It captures all aspects of the system (hardware and software), and encompasses production, disaster recovery and development functionality.

• Please see RackTable for network related documentation. (Things such as port mappings, ip space usage etc). RackTables is the authoritative source, as it's kept up to date via automated scripts.

• Please see our credential management system for access details.

RackTable and Credman are restricted to authorized personnel. This lets us have open documentation, and keep the sensitive bits secure.

Deployed Systems

The physical gear is located in Kansas City in a co location facility by the name of *Joes Data Center* ( http://www.joesdatacenter.com).

The deployed gear is as follows:

• 2 Dell Optiplex 745 (active/passive pfsense routers)
• 2 Cisco 2950 Switch (main/peering switch), slated for HA configuration
• 2 Dell Optiplex 745 (active/passive FreeNAS servers)
• 1 Dell Poweredge 2800 (vm server)
• 2 USB flash drives (root drives)
• 3 USB hard drives (raid and backup data storage)
• DRAC card
• IPMI
• 2 Cyclades PDU (power)

Dell Optiplex 745 (pfsense router)

Specs: Dual Core P4 3.0Ghz / 1 gig

Notes: System can also be accessed via SSH. Not much can be done via SSH unless you know exactly what you are doing.

Cisco 2950 Switch (main/peering switch)

Notes: Please don't do anything on the switch without a full and complete understanding of what you are doing. Under 99.99999% of circumstances, no switch work should need to be done by anyone except Charles.

Specs:

 cisco WS-C2950-24 (RC32300) processor (revision M0) with 20710K bytes of memory.
 Processor board ID FOC0748Y5FT
 Last reset from system-reset
 Running Standard Image
 24 FastEthernet/IEEE 802.3 interface(s)
 
 32K bytes of flash-simulated non-volatile configuration memory.
 Base ethernet MAC Address: 00:0E:83:92:CA:C0
 Motherboard assembly number: 73-5781-12
 Power supply part number: 34-0965-01
 Motherboard serial number: FOC07471LRM
 Power supply serial number: DAB0747GJH9
 Model revision number: M0
 Motherboard revision number: B0
 Model number: WS-C2950-24
 System serial number: FOC0748Y5FT
 Configuration register is 0xF

Dell Poweredge 2800 (vm server)

Access details:

Notes: You can find virtual machine details (name/ip/vlan) in racktable ( http://racktable.freenetworkfoundation.org/index.php?page=object&object_id=1 ).

Specs:

OS:

root@knel-prod-fm1:/data# cat /etc/debian_version 
6.0.3
root@knel-prod-fm1:/data#

root@knel-prod-fm1:/data# uname -a
Linux knel-prod-fm1 2.6.32-5-amd64 #1 SMP Fri Sep 9 20:23:16 UTC 2011 x86_64 GNU/Linux
root@knel-prod-fm1:/data# 

Hardware:
CPU: 2 3.6Ghz dual core Xeon processors (64 bit)
RAM: 6 gigabytes 

See attached dmidecode file for verbose hardware details

PDU

Access details:

Notes: Not hooked to console access yet. All devices are powered through it.

Specs: TBA

DRAC CARD

Notes: Accessed via HTTPS and SSH. Provides reboot functionality and console access. So one can console in (via the web UI or SSH). This will put you at the vm server console. You can use minicom on the vm server to jump to the switch,pfsense,pdu console. You can also reboot the system via the web UI or SSH.

Specs: N/A

IPMI

Notes: Not online yet.

Specs: N/A

Deployed system storage details (hard drive and RAID setup details)

We are using software RAID for the root and /data partition. Both are RAID1. The /backup partition is a single USB drive.

Root

2 8GB USB flash drives Details:

• Overall RAID UUID: /dev/md0: UUID="ab8b199e-2093-499d-8df5-3bedbce1cc7b" TYPE="ext3"
• Raid Member UUID: /dev/sdc1: UUID="7a0c9676-d3b4-6a0d-cd2a-4d3d158dbad1" LABEL="debian:0" TYPE="linux_raid_member"
• Raid Member UUID: /dev/sdb1: UUID="7a0c9676-d3b4-6a0d-cd2a-4d3d158dbad1" LABEL="debian:0" TYPE="linux_raid_member"
• Actual device UUID:
• Actual device UUID:

/data

2 1TB USB hard drives

Overall RAID UUID: /dev/md1: UUID="2e7a169a-c82c-4e92-b6f7-1e8f3c0625f4" TYPE="ext3" 
RAID member UUID:  /dev/sdd1: 06e0cf8-7966-9eca-9dfa-4596c9ac4262 LABEL="debian:1" TYPE="linux_raid_member" 
Actual device UUID: 
Actual device UUID: 

Backup Drive (/backup)

TODO

Misc storage notes

 282  smartctl -i /dev/md0
 283  smartctl -i /dev/sda
 284  smartctl -i /dev/sdb
 285  smartctl -i /dev/sdc
 286  smartctl -i /dev/sdd
 289  smartctl -i /dev/sda
 290  smartctl -i /dev/sdb
 291  smartctl -i /dev/sdc
 292  smartctl -i /dev/sdd
 293  smartctl -i /dev/sde
 294  smartctl -d ata -i /dev/sdd
 295  smartctl -d sat -i /dev/sdd
 296  smartctl -d sat -i /dev/sda
 297  smartctl -d sat -i /dev/sdb
 298  smartctl -d sat -i /dev/sdc
 299  smartctl -d sat -i /dev/sdd
 300  smartctl -d sat -i /dev/sde
 304  smartctl -i /dev/sda
 305  smartctl -d sat -i /dev/sda
 306  smartctl -d sat -i /dev/sdb
 307  smartctl -d sat -i /dev/sdc
 308  smartctl  -i /dev/sdc
 309  smartctl  -i /dev/sdc >> drives 
 310  smartctl -i /dev/sde
 311  smartctl -d sat -i /dev/sde
 312  smartctl -d sat -i /dev/sde >> drives 
 313  smartctl -d sat -i /dev/sdf >> drives 
 344  smartctl -i  /dev/sdb
 346  smartctl -i  /dev/sdc
 347  smartctl -i  /dev/sdc >> drives 
 349  smartctl -i  /dev/sdb
 350  smartctl -d sat -i  /dev/sdb
 351  smartctl -T permissive -d sat -i  /dev/sdb
 352  smartctl -s on -T permissive -d sat -i  /dev/sdb
 356  smartctl -i  /dev/sdd
 357  smartctl -h
 359  smartctl --all /dev/sdc
 360  smartctl --all /dev/sdc >> drives 
 362  smartctl --all /dev/sdd
 363  smartctl --all /dev/sde
 364  smartctl --all /dev/sdf
 365  smartctl -d sat --all /dev/sde
 366  smartctl -d sat --all /dev/sde >> drives 
 368  smartctl -d sat --all /dev/sde >> drives 
 372  smartctl -d sat --all /dev/sdd >> drives 
 374  smartctl -d sat --all /dev/sde
 375  smartctl -d sat --all /dev/sde >> drives 
 377  smartctl -d sat --all /dev/sde 
 378  smartctl -d sat --all /dev/sde >> drives 
 380  smartctl -d sat --all /dev/sdd
 381  smartctl -d sat --all /dev/sdd >> drives 
 383  smartctl -d sat --all /dev/sdc
 384  smartctl  --all /dev/sdc
 385  smartctl  --all /dev/sdc >> drives 
 391  smartctl  --all /dev/sdb
 392  smartctl  -d sat --all /dev/sdb
 394  history |grep smartctl
 395  history |grep smartctl >> drives

Bare metal system software details and virtual machines

• Backup script ``/usr/local/bin/backup.sh`` runs nightly at 4AM EST and backs up /data, cisco, pfsense configuration to the NAS.
• Opsview agent
• OMSA software

OMSA

Notes: Accessed via HTTPS. Provides all manner of system instrumentation data, lets you set IPMI details and other fun stuff.

Specs: N/A

Virtual machines list

root@knel-prod-fm1:~# lxc-ls

• fnf-opsview << opsview server (not running in cPanel VM due to amount of perl in use. Don't think it would play well with cPanel)
• fnf-video << kaltura server (should be migrated to cPanel VM shortly)
• infra-dns << PowerDNS server

• kccp << cPanel VM (all FNF web properties, git, syslog server, freeswitch server, sole HTTP entry point (routes to other servers as needed via mox_proxy)

• fnf-sso-dev << SSO image development VM
• fnf-voiptest << Voip development VM

KC POP - Virtual Machine Creation

The process to create a new virtual machine is pretty straightforward

1. Login to pfsense
2. Go to services -> DHCP Server
3. Add a new DHCP reservation to the appropriate network
4. Login to bare metal server

A virtual machine consists of two components

• Data for the virtual instance: (located in /data/lxc)

kccp
root@knel-prod-fm1:/data/lxc# ls
authorized_keys  fix-dev.sh  infra  occupy     www
fix_dev.sh	 fnf	     knel   templates
root@knel-prod-fm1:/data/lxc# 

root@knel-prod-fm1:/data/lxc# ls fnf/
fnf-base      fnf-freeswitch  fnf-opsview      fnf-sogo
fnf-chili     fnf-git	      fnf-packetfence  fnf-video
fnf-freeside  fnf-logger      fnf-snorby       fnf-voiptest

Configuration files for the virtual instance: (located in /etc/lxc)

root@knel-prod-fm1:/etc/lxc# ls
cnwknel  fnf  infra  knel  occupy  stage  www
root@knel-prod-fm1:/etc/lxc# 

root@knel-prod-fm1:/etc/lxc# ls fnf/
fnf-base.conf	   fnf-freeswitch.conf	fnf-nocproject.conf  fnf-video.conf
fnf-chili.conf	   fnf-git.conf		fnf-opsview.conf     fnf-voiptest.conf
fnf-freeside.conf  fnf-jabber.conf	fnf-sogo.conf
root@knel-prod-fm1:/etc/lxc# 

Creating a new virtual machine is straight forward:

1. Setup the new DHCP lease in PfSense
2. Clone the data directory in /data/lxc/<category> to the new machine name.
3. Clone the config file in /etc/lxc/<category> to the new machine name.
4. Edit the config file and adjust the mac address and data path

 lxc-start -f <path to config> -d -n <name of vm>

Most likely a new VM isn't needed at this point. Cpanel VM should be able to do just about everything LAMP/Java related.

A few notes: The first time you start the new container, you'll want to invoke

 lxc-start -f <path-to-config> -n <name of vm>

(notice no -d)

You'll then be in the container. Make any necessary changes (hostname, password).

 shutdown -h now

will bring you back to the bare metal.

If you'd like to enable ssh access to the new box, there's a couple of extra steps:

• A NAT rule to forward a random port to port 22 on the vm
• A DNS entry to resolve some domain to the correct ip address (this depends on which vlan your vm is on)

KC POP - Network Information

• Public IP information
• domain names / registrar / DNS
• firewall rules
• vpn access

Public IP information

Joes Data Center

• 69.59.131.24/29

ATX

• 68.203.12.180

Public DNS related information

Domain names

• thefnf.org
• freenetworkmovement.org
• freenetworkfoundation.org
• fnf.fm
• fnf.tel
• freenetfound.org

Registrar

Currently it's [1] which has proven very unreliable and we will be moving away from them soon.

New registrar options:

• zoneedit.com
• gandhi

DNS Server

• Primary ns1.thefnf.org/ns3.knownelement.com/69.195.131.30
• Secondary/Tertiary zoneedit

Firewall Rules

PfSense is the authoritative source for firewall rules. This section just provides an overview of the logic behind how they are setup.

VPN

Site to Site VPN

Currently we have the following site to site VPN setup:

• ATX lab (charles house) to FNF primary data center
• LAX lab (josh house) to FNF primary data center
• ATX FT to FNF primary data center
• NYC FT to FNF primary data center

See http://racktable.freenetworkfoundation.org/index.php?page=row&row_id=6 for up to date details, ip space used etc.

Road warrior VPN

This is used for FNF staff when they travel. It allows access to all aspects of the FNF enterprise network.

FNF Web Properties

FNF Enterprise Infrastructure

These are applications which support FNF business operations, project management, infrastructure operations and documentation. All of the below software is deployed on our cPanel VM unless otherwise noted.

The majority of the applications listed below are in production status. A few are still in the early stages (and are marked as such), but should be finished in the near future.

What	Software	Deployment
blog and main site	wordpress	http://www.freenetworkfoundation.org
Project management	Chiliproject	http://chili.freenetworkfoundation.org/
Privileged credential management	TeamPass	http://credman.freenetworkfoundation.org/
crm	civicrm which is drupal based	http://crm.freenetworkfoundation.org/
Document management	OpenDocMan	http://documents.freenetworkfoundation.org/
Real time shared text editing	Etherpad	http://etherpad.freenetworkfoundation.org
Voice/Video communications	Whistle/BlueBox from 2600hz.org	Not currently in an operational/working state.
HR system	OrangeHRM	http://hr.freenetworkfoundation.org/
Finance/accounting system	Dolibar	http://incbits.freenetworkfoundation.org/
Brand tracking, relevant topic tracking	Tattler	http://legwatch.freenetworkfoundation.org/
Learning management system	Canvas	http://learn.freenetworkfoundation.org
Local chapter meeting system	WebCalendar	http://meetings.freenetworkfoundation.org
Microblogging	Status.net	http://mblog.freenetworkfoundation.org/
Mailing/campaign manager	phpList	http://phplist.freenetworkfoundation.org
Photo gallery	Piwigo	http://photos.freenetworkfoundation.org/
Social networking insight	Thinkup	http://snetinsight.freenetworkfoundation.org/
FNF Social Network	Lorea	http://social.freenetworkfoundation.org
Web analytics	Piwik	http://webstats.freenetworkfoundation.org/
wiki	media wiki	http://www.freenetworkmovement.org/commons
IDS alerting/management	Snorby	http://snorby.freenetworkfoundation.org
System change tracking	OSSEC	http://ossec.freenetworkfoundation.org
Log insight	Octopussy	Not currently in an operational/working state.
Virtual machine management	OpenVZ Web Panel	http://vzmanage.freenetworkfoundation.org
Text chat	Openfire	http://openfire.freenetworkfoundation.org
Internal DNS	PowerDNS (hosting internal domains)	http://pdns.freenetworkfoundation.org/padmin (cPanel VM proxy back to running on 10.250.6.5)
Source control	GIT	This is deployed in conjunction with chili.
Performance optimization	memcached	Installed (with pecl extension), not currently in use but available. In particular the wiki and blog can take immediate advantage if needed.

Disaster recovery

We have a 1U server in Dallas TX handling disaster recovery functionality for FNF NOC and enterprise services (everything on this page).

Operator Support System tools

Under heavy construction at this time!

All applications below are subject to being replaced, having data wiped etc at any time. Once my CIO tasks are finished, I'll be doing a very thorough build out on the NOC.

What	Software	Deployment
Ticketing	OSTicket	http://support.freenetworkfoundation.org
Monitoring	Opsview	http://opsview.freenetworkfoundation.org/
Documentation and configuration management system	Netdot/Nocproject.org	Not yet deployed.
Access point management management	Aircontrol	http://aircontrol.freenetworkfoundation.org/home.seam
GIS System	Udig/OpenStreetMap/Ushadi	http://ushadi.freenetworkfoundation.org/
Asset location tracking	OpenGTS	Not yet deployed
Technical Operations Dispatch system	TicketsCAD	http://dispatch.freenetworkfoundation.org/
Knowledge base	phpMyFAQ	http://kb.freenetworkfoundation.org/
Colo documentation	rack table	http://racktable.freenetworkfoundation.org/
Network diagramming tool	netdiag	http://netmap.freenetworkfoundation.org/

KC/ATX Lab

See FreedomLab for all things related to the lab. Racktables also has a lot of info.

Dallas DR

This is our disaster recovery site. We have a single 1U server running several virtual machines.

More details later.