# Magento Enterprise 1.12 cluster

[Scaling Magento presentation with 60 slides and notes](https://docs.google.com/a/copiousinc.com/presentation/d/1rHM7LOfzgfol6FEijLjlCpTBPaYJlMbI-WPpZL59Dng/edit), which was then [reviewed by Alan Storm](http://alanstorm.com/scaling_magento_at_copious).

## Table of Contents

* [Summary](#summary)
* [August 2013 traffic](#august-2013-traffic)
* [Web hosts](#web-hosts)
    * [Performance](#performance)
    * [Configuration](#modules)
* [Cache hosts](#cache-hosts)
    * [Performance](#performance)
    * [Sessions](#sessions)
    * [Backend](#backend)
    * [FPC](#fpc)
    * [Configuration](#configuration)
* [Database hosts](#database-hosts)
    * [Performance](#performance)
    * [Statistics](#statistics)
    * [Configuration](#configuration)
* [Utility hosts](#utility-hosts)

------

## Summary

* Peak capacity of 100,000 orders in a day (4,000 an hour or 80/minute burst)
* 806,000 lines of code (100,000 lines more than core + Zend)
* 97% PHP
* 1.6% Ruby
* 0.45% Perl
* 0.28% Shell
* 0.12% Python
* Admin users: 46
* Nearly 2,000 internal users (managers and employees)
* Categories: 2,450 (largest one has 2,400 products)
* Product entities: 101,000
* Combo products: 484
* Product relations: 54,000
* In stock and enabled configurable products: 10,100
* CMS blocks: 3,100
* CMS pages: 1,400

## August 2013 traffic

* 40 million monthly pageviews
* 2.3 million unique visitors
* 46,000 monthly checkouts
* 89% of visitors from the USA
* 34% of visits from iOS
* 14% of visits from Android

## Web hosts

There are 10 hosts behind redundant, highly available hardware firewalls and hardware load balancers.

### Performance

Over the first 90 days:

* site-wide average response time: 282 ms
* load average: 0.6 to 1.0 (in tests, performance degrades by 35% when load averages hit ~5.0)
* CPU utilization averages 14%
* disk IO averages 3%
* network throughput averages 5 mbps transmitted and 17 mbps received

### Configuration

* Ubuntu 12.04 LTS 64-bit
* 3.5.0-36-generic GNU/Linux kernel
* Dual Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (4 cores each)
* 32 GB DDR3 1333 MHz RAM
* PHP-FPM, 5.3.10-1ubuntu3.7

### Modules

* https://github.com/colinmollenhour/Cm_RedisSession
* https://github.com/colinmollenhour/Cm_Cache_Backend_Redis
* https://github.com/aligent/CacheObserver.git --branch=models
* https://github.com/aligent/Lokey_ShippingAdjustments
* https://github.com/aligent/VF_CustomMenu.git --branch=production
* https://github.com/Vinai/customer-activation
* https://github.com/gknoppe-guidance/magento-cachebuster

## Cache hosts

There are two hosts running Redis in a master-slave configuration with automated failover.

Three Redis instances are used to increase throughput and provide fine-tuning of persistence behaviors.

### Performance

Over the first 90 days:

* 3,000 commands per second
* 0.7 ms average response time
* load average of 1.0 to 1.5
* CPU utilization averages 7%
* disk IO averages 11%
* network throughput averages 150 mbps transmitted and 30 mbps received

### Sessions

* average of 6–8 million keys (20–26 GB)
* commands per second: ~400 (peaking at 1,500 under load)
* expirations per second: ~40

### Backend

* average of ~1 million keys (6–10 GB)
* commands per second: ~3,250 (peaking at 5,000 under load)
* expirations per second: ~100
* hit rate: ~85%

### FPC

* average of ~450,000 keys (4–8 GB)
* commands per second: ~2,000 (peaking at 5,000 under load)
* expirations per second: ~10
* hit rate: ~80%

### Configuration

* Ubuntu 12.04 LTS 64-bit
* 3.2.0-48-generic GNU/Linux kernel
* Quad Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (6 cores each)
* 128 GB buffered DDR3 1333 MHz RAM
* Mechanical disks, RAID 1, hardware controller
* Redis 2.6.11

## Database hosts

There are two hosts running MySQL 5.6.11 in a master-slave configuration with warm failover.


### Performance

Over the first 90 days:

* 1,500 commands per second
* 1.1 ms average response time
* load average of 0.1 (master) and 0.4 (slave)
* CPU utilization averages 0% (master) and 1.5% (slave)
* disk IO averages 1% (master) and 0.5% (slave)
* network throughput averages 40 mbps transmitted (slave) and 0.7 mbps received (master)

### Statistics

Over the first 100 days:

* 42 TB of transmitted data
* 23 TB of innodb writes
* 8.4 TB of innodb log churn
* 100% thread cache hit rate
* 99.996% table cache hit rate (large number of open tables possibly related to MySQL bugs #16244691 and #65384)
* 99.9999993% of table locks are immediate (most are nightly processes)
* Innodb_buffer_pool_wait_free: 0
* Innodb_log_waits: 0
* 85% query cache hit rate; this doesn’t really mean anything with such high churn rate of data
* 94.5% of temp tables are in memory (only nightly processes require disk tables)
* 99.97% of queries are faster than 200 ms; we’ve reached a plateau of optimization
* The average row lock is a bit slow as a consequence of Magento indexing architecture and our background processing queues
* Moderate rate of random and sequential reads (table scans!), but we can absorb that overhead with hardware and focus on improving PHP code


### Configuration

Host:

* Ubuntu 12.04 LTS 64-bit
* 3.5.0-34-generic GNU/Linux
* Quad Intel(R) Xeon(R) CPU E7- 2860  @ 2.27GHz (10 cores each)
* 128 GB buffered DDR3 1333 MHz RAM
* SSD, RAID 1+0, hardware controller
* MySQL 5.6.11 with tcmalloc

See also: [MySQL configurations](#file-data-etc-mysql-my-cnf)


## Utility hosts

These run cron and systems tasks. Daily backups are stored offsite and verified functional! :)

Scripts are in: Bash, Ruby, Perl, Python, and PHP.

Chef Server 10 for deployments and configuration management. Database tests, using Perl’s Test::DatabaseRow, are used to catch some human errors.