TechOPS

TechnicalOperations

All things related to FNF Technical Operations. This covers all three FNF locations (MCI,AUS,DFW).

Kansas City Point of Presence - Data center documentation

The purpose of this section is to provide documentation of the FNF enterprise infrastructure deployed in Kansas City. It captures all aspects of the system (hardware and software), and encompasses production, disaster recovery and development functionality.

• Please see RackTable for network related documentation. (Things such as port mappings, ip space usage etc). RackTables is the authoritative source, as it's kept up to date via automated scripts.

• Please see our credential management system for access details.

RackTable and Credman are restricted to authorized personnel. This lets us have open documentation, and keep the sensitive bits secure.

Deployed Systems

The physical gear is located in Kansas City in a co location facility by the name of *Joes Data Center* ( http://www.joesdatacenter.com).

The deployed gear is as follows:

• 2 Dell Optiplex 745 (active/passive pfsense routers)
• 2 Cisco 2950 Switch (main/peering switch), slated for HA configuration
• 2 Dell Optiplex 745 (active/passive FreeNAS servers)
• 1 Dell Poweredge 2800 (vm server)
• 2 USB flash drives (root drives)
• 3 USB hard drives (raid and backup data storage)
• DRAC card
• IPMI
• 2 Cyclades PDU (power)

Dell Optiplex 745 (pfsense router)

• Specs:*

Dual Core P4 3.0Ghz / 1 gig

• Notes:*

System can also be accessed via SSH. Not much can be done via SSH unless you know exactly what you are doing.

Cisco 2950 Switch (main/peering switch)

• Notes:*

Please don't do anything on the switch without a full and complete understanding of what you are doing. Under 99.99999% of circumstances, no switch work should need to be done by anyone except Charles.

• Specs:*

cisco WS-C2950-24 (RC32300) processor (revision M0) with 20710K bytes of memory. Processor board ID FOC0748Y5FT Last reset from system-reset Running Standard Image 24 FastEthernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address: 00:0E:83:92:CA:C0 Motherboard assembly number: 73-5781-12 Power supply part number: 34-0965-01 Motherboard serial number: FOC07471LRM Power supply serial number: DAB0747GJH9 Model revision number: M0 Motherboard revision number: B0 Model number: WS-C2950-24 System serial number: FOC0748Y5FT Configuration register is 0xF

Dell Poweredge 2800 (vm server)

• Access details*:

• Notes:*

You can find virtual machine details (name/ip/vlan) in racktable ( http://racktable.freenetworkfoundation.org/index.php?page=object&object_id=1 ).

• Specs:*

OS:

root@knel-prod-fm1:/data# cat /etc/debian_version 6.0.3 root@knel-prod-fm1:/data#

root@knel-prod-fm1:/data# uname -a Linux knel-prod-fm1 2.6.32-5-amd64 #1 SMP Fri Sep 9 20:23:16 UTC 2011 x86_64 GNU/Linux root@knel-prod-fm1:/data#

Hardware: CPU: 2 3.6Ghz dual core Xeon processors (64 bit) RAM: 6 gigabytes

See attached dmidecode file for verbose hardware details

PDU

• Access details:*

• Notes:*

Not hooked to console access yet. All devices are powered through it.

• Specs:*

TBA

DRAC CARD

• Notes:*

Accessed via HTTPS and SSH. Provides reboot functionality and console access. So one can console in (via the web UI or SSH). This will put you at the vm server console. You can use minicom on the vm server to jump to the switch,pfsense,pdu console. You can also reboot the system via the web UI or SSH.

• Specs:*

N/A

IPMI

• Notes:*

Not online yet.

• Specs: *

N/A

h2. Deployed system storage details (hard drive and RAID setup details)

We are using software RAID for the root and /data partition. Both are RAID1. The /backup partition is a single USB drive.

h3. *Root*

2 8GB USB flash drives Details:

• Overall RAID UUID: /dev/md0: UUID="ab8b199e-2093-499d-8df5-3bedbce1cc7b" TYPE="ext3"
• Raid Member UUID: /dev/sdc1: UUID="7a0c9676-d3b4-6a0d-cd2a-4d3d158dbad1" LABEL="debian:0" TYPE="linux_raid_member"
• Raid Member UUID: /dev/sdb1: UUID="7a0c9676-d3b4-6a0d-cd2a-4d3d158dbad1" LABEL="debian:0" TYPE="linux_raid_member"
• Actual device UUID:
• Actual device UUID:

h3. */data*

2 1TB USB hard drives Overall RAID UUID: /dev/md1: UUID="2e7a169a-c82c-4e92-b6f7-1e8f3c0625f4" TYPE="ext3" RAID member UUID: /dev/sdd1: 06e0cf8-7966-9eca-9dfa-4596c9ac4262 LABEL="debian:1" TYPE="linux_raid_member" Actual device UUID: Actual device UUID:

h3. Backup Drive (/backup)

TODO

h3. Misc storage notes:

 282  smartctl -i /dev/md0
 283  smartctl -i /dev/sda
 284  smartctl -i /dev/sdb
 285  smartctl -i /dev/sdc
 286  smartctl -i /dev/sdd
 289  smartctl -i /dev/sda
 290  smartctl -i /dev/sdb
 291  smartctl -i /dev/sdc
 292  smartctl -i /dev/sdd
 293  smartctl -i /dev/sde
 294  smartctl -d ata -i /dev/sdd
 295  smartctl -d sat -i /dev/sdd
 296  smartctl -d sat -i /dev/sda
 297  smartctl -d sat -i /dev/sdb
 298  smartctl -d sat -i /dev/sdc
 299  smartctl -d sat -i /dev/sdd
 300  smartctl -d sat -i /dev/sde
 304  smartctl -i /dev/sda
 305  smartctl -d sat -i /dev/sda
 306  smartctl -d sat -i /dev/sdb
 307  smartctl -d sat -i /dev/sdc
 308  smartctl  -i /dev/sdc
 309  smartctl  -i /dev/sdc >> drives 
 310  smartctl -i /dev/sde
 311  smartctl -d sat -i /dev/sde
 312  smartctl -d sat -i /dev/sde >> drives 
 313  smartctl -d sat -i /dev/sdf >> drives 
 344  smartctl -i  /dev/sdb
 346  smartctl -i  /dev/sdc
 347  smartctl -i  /dev/sdc >> drives 
 349  smartctl -i  /dev/sdb
 350  smartctl -d sat -i  /dev/sdb
 351  smartctl -T permissive -d sat -i  /dev/sdb
 352  smartctl -s on -T permissive -d sat -i  /dev/sdb
 356  smartctl -i  /dev/sdd
 357  smartctl -h
 359  smartctl --all /dev/sdc
 360  smartctl --all /dev/sdc >> drives 
 362  smartctl --all /dev/sdd
 363  smartctl --all /dev/sde
 364  smartctl --all /dev/sdf
 365  smartctl -d sat --all /dev/sde
 366  smartctl -d sat --all /dev/sde >> drives 
 368  smartctl -d sat --all /dev/sde >> drives 
 372  smartctl -d sat --all /dev/sdd >> drives 
 374  smartctl -d sat --all /dev/sde
 375  smartctl -d sat --all /dev/sde >> drives 
 377  smartctl -d sat --all /dev/sde 
 378  smartctl -d sat --all /dev/sde >> drives 
 380  smartctl -d sat --all /dev/sdd
 381  smartctl -d sat --all /dev/sdd >> drives 
 383  smartctl -d sat --all /dev/sdc
 384  smartctl  --all /dev/sdc
 385  smartctl  --all /dev/sdc >> drives 
 391  smartctl  --all /dev/sdb
 392  smartctl  -d sat --all /dev/sdb
 394  history |grep smartctl
 395  history |grep smartctl >> drives

Bare metal system software details and virtual machines

• Backup script */usr/local/bin/backup.sh* runs nightly at 4AM EST and backs up /data, cisco, pfsense configuration to the NAS.
• Opsview agent
• OMSA software

OMSA

• Notes:*

Accessed via HTTPS. Provides all manner of system instrumentation data, lets you set IPMI details and other fun stuff.

• Specs:*

N/A

Virtual machines list

root@knel-prod-fm1:~# lxc-ls

• fnf-opsview << opsview server (not running in cPanel VM due to amount of perl in use. Don't think it would play well with cPanel)
• fnf-video << kaltura server (should be migrated to cPanel VM shortly)
• infra-dns << PowerDNS server

• kccp << cPanel VM (all FNF web properties, git, syslog server, freeswitch server, sole HTTP entry point (routes to other servers as needed via mox_proxy)

• fnf-sso-dev << SSO image development VM
• fnf-voiptest << Voip development VM

root@knel-prod-fm1:~#

FreedomNoc software (this encompasses both FNF enterprise software and operator facing software)

See http://chili.freenetworkfoundation.org/projects/freedomnoc/wiki/DeployedCapabilities for details.

h2. Misc notes to integrate later

• http://www.tylerdave.com/2009/10/12/configuring-passenger-on-cpanel-systems/
• http://docs.cpanel.net/twiki/pub/AllDocumentation/TrainingResources/TrainingSlides08/Tomcat.pdf