View source for TechOPS

=TechnicalOperations=

All things related to FNF Technical Operations. This covers all three FNF locations (MCI,AUS,DFW).

==Kansas City Point of Presence - Data center documentation==

The purpose of this section is to provide documentation of the FNF enterprise infrastructure deployed in Kansas City. It captures all aspects of the system (hardware and software), and encompasses production, disaster recovery and development functionality.

* Please see [http://racktable.freenetworkfoundation.org/ RackTable] for network related documentation. (Things such as port mappings, ip space usage etc). RackTables is the authoritative source, as it's kept up to date via automated scripts. 

* Please see our [http://credman.freenetworkfoundation.org credential management system] for access details. 

RackTable and Credman are restricted to authorized personnel. This lets us have open documentation, and keep the sensitive bits secure. 

===Deployed Systems===

The physical gear is located in Kansas City in a co location facility by the name of *Joes Data Center* ( http://www.joesdatacenter.com). 

The deployed gear is as follows:

* 2 Dell Optiplex 745 (active/passive pfsense routers)
* 2 Cisco 2950 Switch (main/peering switch), slated for HA configuration
* 2 Dell Optiplex 745 (active/passive FreeNAS servers)
* 1 Dell Poweredge 2800 (vm server)
* 2 USB flash drives  (root drives)
* 3 USB hard drives (raid and backup data storage)
* DRAC card
* IPMI
* 2 Cyclades PDU (power)

====Dell Optiplex 745 (pfsense router)====
*Specs:*
Dual Core P4 3.0Ghz / 1 gig

*Notes:*
System can also be accessed via SSH. Not much can be done via SSH unless you know exactly what you are doing. 


====Cisco 2950 Switch (main/peering switch)====

*Notes:*

Please don't do anything on the switch without a full and complete understanding of what you are doing. Under 99.99999% of circumstances, no switch work should need to be done by anyone except Charles.

*Specs:*

cisco WS-C2950-24 (RC32300) processor (revision M0) with 20710K bytes of memory.
Processor board ID FOC0748Y5FT
Last reset from system-reset
Running Standard Image
24 FastEthernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:0E:83:92:CA:C0
Motherboard assembly number: 73-5781-12
Power supply part number: 34-0965-01
Motherboard serial number: FOC07471LRM
Power supply serial number: DAB0747GJH9
Model revision number: M0
Motherboard revision number: B0
Model number: WS-C2950-24
System serial number: FOC0748Y5FT
Configuration register is 0xF


====Dell Poweredge 2800 (vm server)====

*Access details*:


*Notes:*

You can find virtual machine details (name/ip/vlan) in racktable ( http://racktable.freenetworkfoundation.org/index.php?page=object&object_id=1 ).

*Specs:*

OS:

root@knel-prod-fm1:/data# cat /etc/debian_version 
6.0.3
root@knel-prod-fm1:/data#

root@knel-prod-fm1:/data# uname -a
Linux knel-prod-fm1 2.6.32-5-amd64 #1 SMP Fri Sep 9 20:23:16 UTC 2011 x86_64 GNU/Linux
root@knel-prod-fm1:/data# 

Hardware:
CPU: 2 3.6Ghz dual core Xeon processors (64 bit)
RAM: 6 gigabytes 

See attached dmidecode file for verbose hardware details


====PDU====

*Access details:*

*Notes:* 
Not hooked to console access yet. All devices are powered through it.

*Specs:*
TBA

====DRAC CARD====


*Notes:*
Accessed via HTTPS and SSH. Provides reboot functionality and console access. So one can console in (via the web UI or SSH). This will put you at the vm server console.
You can use minicom on the vm server to jump to the switch,pfsense,pdu console. You can also reboot the system via the web UI or SSH. 

*Specs:*
N/A

====IPMI====


*Notes:*

Not online yet. 

*Specs: *
N/A

h2. Deployed system storage details (hard drive and RAID setup details)

We are using software RAID for the root and /data partition. Both are RAID1. The /backup partition is a single USB drive.


h3. *Root* 

2 8GB USB flash drives
Details:

* Overall RAID UUID: /dev/md0: UUID="ab8b199e-2093-499d-8df5-3bedbce1cc7b" TYPE="ext3" 
* Raid Member UUID: /dev/sdc1: UUID="7a0c9676-d3b4-6a0d-cd2a-4d3d158dbad1" LABEL="debian:0" TYPE="linux_raid_member" 
* Raid Member UUID: /dev/sdb1: UUID="7a0c9676-d3b4-6a0d-cd2a-4d3d158dbad1" LABEL="debian:0" TYPE="linux_raid_member" 
* Actual device UUID: 
* Actual device UUID:

h3. */data*

2 1TB USB hard drives
Overall RAID UUID: /dev/md1: UUID="2e7a169a-c82c-4e92-b6f7-1e8f3c0625f4" TYPE="ext3" 
RAID member UUID:  /dev/sdd1: 06e0cf8-7966-9eca-9dfa-4596c9ac4262 LABEL="debian:1" TYPE="linux_raid_member" 
Actual device UUID: 
Actual device UUID: 

h3. Backup Drive (/backup)

TODO


h3. Misc storage notes:

  282  smartctl -i /dev/md0
  283  smartctl -i /dev/sda
  284  smartctl -i /dev/sdb
  285  smartctl -i /dev/sdc
  286  smartctl -i /dev/sdd
  289  smartctl -i /dev/sda
  290  smartctl -i /dev/sdb
  291  smartctl -i /dev/sdc
  292  smartctl -i /dev/sdd
  293  smartctl -i /dev/sde
  294  smartctl -d ata -i /dev/sdd
  295  smartctl -d sat -i /dev/sdd
  296  smartctl -d sat -i /dev/sda
  297  smartctl -d sat -i /dev/sdb
  298  smartctl -d sat -i /dev/sdc
  299  smartctl -d sat -i /dev/sdd
  300  smartctl -d sat -i /dev/sde
  304  smartctl -i /dev/sda
  305  smartctl -d sat -i /dev/sda
  306  smartctl -d sat -i /dev/sdb
  307  smartctl -d sat -i /dev/sdc
  308  smartctl  -i /dev/sdc
  309  smartctl  -i /dev/sdc >> drives 
  310  smartctl -i /dev/sde
  311  smartctl -d sat -i /dev/sde
  312  smartctl -d sat -i /dev/sde >> drives 
  313  smartctl -d sat -i /dev/sdf >> drives 
  344  smartctl -i  /dev/sdb
  346  smartctl -i  /dev/sdc
  347  smartctl -i  /dev/sdc >> drives 
  349  smartctl -i  /dev/sdb
  350  smartctl -d sat -i  /dev/sdb
  351  smartctl -T permissive -d sat -i  /dev/sdb
  352  smartctl -s on -T permissive -d sat -i  /dev/sdb
  356  smartctl -i  /dev/sdd
  357  smartctl -h
  359  smartctl --all /dev/sdc
  360  smartctl --all /dev/sdc >> drives 
  362  smartctl --all /dev/sdd
  363  smartctl --all /dev/sde
  364  smartctl --all /dev/sdf
  365  smartctl -d sat --all /dev/sde
  366  smartctl -d sat --all /dev/sde >> drives 
  368  smartctl -d sat --all /dev/sde >> drives 
  372  smartctl -d sat --all /dev/sdd >> drives 
  374  smartctl -d sat --all /dev/sde
  375  smartctl -d sat --all /dev/sde >> drives 
  377  smartctl -d sat --all /dev/sde 
  378  smartctl -d sat --all /dev/sde >> drives 
  380  smartctl -d sat --all /dev/sdd
  381  smartctl -d sat --all /dev/sdd >> drives 
  383  smartctl -d sat --all /dev/sdc
  384  smartctl  --all /dev/sdc
  385  smartctl  --all /dev/sdc >> drives 
  391  smartctl  --all /dev/sdb
  392  smartctl  -d sat --all /dev/sdb
  394  history |grep smartctl
  395  history |grep smartctl >> drives

===Bare metal system software details and virtual machines===

* Backup script */usr/local/bin/backup.sh* runs nightly at 4AM EST and backs up /data, cisco, pfsense configuration to the NAS.
* Opsview agent
* OMSA software

====OMSA====


*Notes:*
Accessed via HTTPS. Provides all manner of system instrumentation data, lets you set IPMI details and other fun stuff. 

*Specs:*
N/A

====Virtual machines list====

root@knel-prod-fm1:~# lxc-ls

* fnf-opsview  << opsview server (not running in cPanel VM due to amount of perl in use. Don't think it would play well with cPanel)
* fnf-video  << kaltura server (should be migrated to cPanel VM shortly)
* infra-dns << PowerDNS server

* kccp << cPanel VM (all FNF web properties, git, syslog server, freeswitch server, sole HTTP entry point (routes to other servers as needed via mox_proxy)

* fnf-sso-dev  << SSO image development VM
* fnf-voiptest << Voip development VM

root@knel-prod-fm1:~#

===FreedomNoc software (this encompasses both FNF enterprise software and operator facing software)===

See http://chili.freenetworkfoundation.org/projects/freedomnoc/wiki/DeployedCapabilities for details. 

 
h2. Misc notes to integrate later

* http://www.tylerdave.com/2009/10/12/configuring-passenger-on-cpanel-systems/
* http://docs.cpanel.net/twiki/pub/AllDocumentation/TrainingResources/TrainingSlides08/Tomcat.pdf

==KC Lab==

===Network Systems===

===Server systems===

==ATX Lab==

===Network Systems===

===Server systems===

==Dallas DR==