Skip to content

Instantly share code, notes, and snippets.

@akanto

akanto/perf-test-repl3

Created April 11, 2016 14:47
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save akanto/3c353a6f1bbc14781a6a3a0929e38cfa to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save akanto/3c353a6f1bbc14781a6a3a0929e38cfa to your computer and use it in GitHub Desktop.
Download ZIP
perf-test-repl3
Raw
perf-test-repl3
[hdfs@docker-ambari ~]# for i in 1 4 8 12 16 20 24 28 32 36; do
> hadoop jar $JAR_JOBCLIENT TestDFSIO -Ddfs.replication=3 -write -nrFiles $i -fileSize 8192
> done
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:03:40 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:03:40 INFO fs.TestDFSIO: nrFiles = 1
16/04/11 14:03:40 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:03:40 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:03:40 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:03:41 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 1 files
16/04/11 14:03:42 INFO fs.TestDFSIO: created control files for: 1 files
16/04/11 14:03:42 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:03:42 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:03:43 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:03:43 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:03:43 INFO mapred.FileInputFormat: Total input paths to process : 1
16/04/11 14:03:43 INFO mapreduce.JobSubmitter: number of splits:1
16/04/11 14:03:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0011
16/04/11 14:03:43 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0011
16/04/11 14:03:43 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0011/
16/04/11 14:03:43 INFO mapreduce.Job: Running job: job_1460382022143_0011
16/04/11 14:03:50 INFO mapreduce.Job: Job job_1460382022143_0011 running in uber mode : false
16/04/11 14:03:50 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:04:01 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:04:29 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 14:04:34 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:04:34 INFO mapreduce.Job: Job job_1460382022143_0011 completed successfully
16/04/11 14:04:34 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=93
FILE: Number of bytes written=266912
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=256
HDFS: Number of bytes written=8589934669
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=35620
Total time spent by all reduces in occupied slots (ms)=5714
Total time spent by all map tasks (ms)=35620
Total time spent by all reduce tasks (ms)=2857
Total vcore-seconds taken by all map tasks=35620
Total vcore-seconds taken by all reduce tasks=2857
Total megabyte-seconds taken by all map tasks=127662080
Total megabyte-seconds taken by all reduce tasks=20478976
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=77
Map output materialized bytes=93
Input split bytes=144
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=93
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=798
CPU time spent (ms)=30310
Physical memory (bytes) snapshot=2856693760
Virtual memory (bytes) snapshot=10525786112
Total committed heap usage (bytes)=5114953728
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=112
File Output Format Counters
Bytes Written=77
16/04/11 14:04:34 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:04:34 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:04:34 UTC 2016
16/04/11 14:04:34 INFO fs.TestDFSIO: Number of files: 1
16/04/11 14:04:34 INFO fs.TestDFSIO: Total MBytes processed: 8192.0
16/04/11 14:04:34 INFO fs.TestDFSIO: Throughput mb/sec: 256.45681369940206
16/04/11 14:04:34 INFO fs.TestDFSIO: Average IO rate mb/sec: 256.4568176269531
16/04/11 14:04:34 INFO fs.TestDFSIO: IO rate std deviation: 0.04748847970735138
16/04/11 14:04:34 INFO fs.TestDFSIO: Test exec time sec: 51.981
16/04/11 14:04:34 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:04:35 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:04:35 INFO fs.TestDFSIO: nrFiles = 4
16/04/11 14:04:35 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:04:35 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:04:35 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:04:36 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 4 files
16/04/11 14:04:36 INFO fs.TestDFSIO: created control files for: 4 files
16/04/11 14:04:37 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:04:37 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:04:37 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:04:37 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:04:38 INFO mapred.FileInputFormat: Total input paths to process : 4
16/04/11 14:04:38 INFO mapreduce.JobSubmitter: number of splits:4
16/04/11 14:04:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0012
16/04/11 14:04:38 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0012
16/04/11 14:04:38 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0012/
16/04/11 14:04:38 INFO mapreduce.Job: Running job: job_1460382022143_0012
16/04/11 14:04:44 INFO mapreduce.Job: Job job_1460382022143_0012 running in uber mode : false
16/04/11 14:04:44 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:04:55 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:05:29 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 14:05:30 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 14:05:34 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:05:35 INFO mapreduce.Job: Job job_1460382022143_0012 completed successfully
16/04/11 14:05:35 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=361
FILE: Number of bytes written=667638
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1024
HDFS: Number of bytes written=34359738450
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Job Counters
Launched map tasks=4
Launched reduce tasks=1
Data-local map tasks=4
Total time spent by all maps in occupied slots (ms)=175180
Total time spent by all reduces in occupied slots (ms)=5254
Total time spent by all map tasks (ms)=175180
Total time spent by all reduce tasks (ms)=2627
Total vcore-seconds taken by all map tasks=175180
Total vcore-seconds taken by all reduce tasks=2627
Total megabyte-seconds taken by all map tasks=627845120
Total megabyte-seconds taken by all reduce tasks=18830336
Map-Reduce Framework
Map input records=4
Map output records=20
Map output bytes=315
Map output materialized bytes=379
Input split bytes=576
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=379
Reduce input records=20
Reduce output records=5
Spilled Records=40
Shuffled Maps =4
Failed Shuffles=0
Merged Map outputs=4
GC time elapsed (ms)=2455
CPU time spent (ms)=121620
Physical memory (bytes) snapshot=10513453056
Virtual memory (bytes) snapshot=21625536512
Total committed heap usage (bytes)=14103347200
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=448
File Output Format Counters
Bytes Written=82
16/04/11 14:05:35 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:05:35 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:05:35 UTC 2016
16/04/11 14:05:35 INFO fs.TestDFSIO: Number of files: 4
16/04/11 14:05:35 INFO fs.TestDFSIO: Total MBytes processed: 32768.0
16/04/11 14:05:35 INFO fs.TestDFSIO: Throughput mb/sec: 204.28545600768066
16/04/11 14:05:35 INFO fs.TestDFSIO: Average IO rate mb/sec: 204.29318237304688
16/04/11 14:05:35 INFO fs.TestDFSIO: IO rate std deviation: 1.2603986048110347
16/04/11 14:05:35 INFO fs.TestDFSIO: Test exec time sec: 59.029
16/04/11 14:05:35 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:05:37 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:05:37 INFO fs.TestDFSIO: nrFiles = 8
16/04/11 14:05:37 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:05:37 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:05:37 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:05:38 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 8 files
16/04/11 14:05:38 INFO fs.TestDFSIO: created control files for: 8 files
16/04/11 14:05:39 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:05:39 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:05:39 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:05:39 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:05:39 INFO mapred.FileInputFormat: Total input paths to process : 8
16/04/11 14:05:39 INFO mapreduce.JobSubmitter: number of splits:8
16/04/11 14:05:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0013
16/04/11 14:05:40 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0013
16/04/11 14:05:40 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0013/
16/04/11 14:05:40 INFO mapreduce.Job: Running job: job_1460382022143_0013
16/04/11 14:05:46 INFO mapreduce.Job: Job job_1460382022143_0013 running in uber mode : false
16/04/11 14:05:46 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:05:56 INFO mapreduce.Job: map 17% reduce 0%
16/04/11 14:05:57 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:07:02 INFO mapreduce.Job: map 71% reduce 0%
16/04/11 14:07:03 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 14:07:05 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 14:07:06 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 14:07:07 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:07:07 INFO mapreduce.Job: Job job_1460382022143_0013 completed successfully
16/04/11 14:07:07 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=715
FILE: Number of bytes written=1201927
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2048
HDFS: Number of bytes written=68719476816
HDFS: Number of read operations=35
HDFS: Number of large read operations=0
HDFS: Number of write operations=10
Job Counters
Launched map tasks=8
Launched reduce tasks=1
Data-local map tasks=8
Total time spent by all maps in occupied slots (ms)=620307
Total time spent by all reduces in occupied slots (ms)=5460
Total time spent by all map tasks (ms)=620307
Total time spent by all reduce tasks (ms)=2730
Total vcore-seconds taken by all map tasks=620307
Total vcore-seconds taken by all reduce tasks=2730
Total megabyte-seconds taken by all map tasks=2223180288
Total megabyte-seconds taken by all reduce tasks=19568640
Map-Reduce Framework
Map input records=8
Map output records=40
Map output bytes=629
Map output materialized bytes=757
Input split bytes=1152
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=757
Reduce input records=40
Reduce output records=5
Spilled Records=80
Shuffled Maps =8
Failed Shuffles=0
Merged Map outputs=8
GC time elapsed (ms)=5616
CPU time spent (ms)=266130
Physical memory (bytes) snapshot=20570271744
Virtual memory (bytes) snapshot=36384845824
Total committed heap usage (bytes)=25811746816
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=896
File Output Format Counters
Bytes Written=80
16/04/11 14:07:07 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:07:07 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:07:07 UTC 2016
16/04/11 14:07:07 INFO fs.TestDFSIO: Number of files: 8
16/04/11 14:07:07 INFO fs.TestDFSIO: Total MBytes processed: 65536.0
16/04/11 14:07:07 INFO fs.TestDFSIO: Throughput mb/sec: 111.38417819836907
16/04/11 14:07:07 INFO fs.TestDFSIO: Average IO rate mb/sec: 111.43865203857422
16/04/11 14:07:07 INFO fs.TestDFSIO: IO rate std deviation: 2.4896417815778475
16/04/11 14:07:07 INFO fs.TestDFSIO: Test exec time sec: 89.212
16/04/11 14:07:07 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:07:09 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:07:09 INFO fs.TestDFSIO: nrFiles = 12
16/04/11 14:07:09 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:07:09 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:07:09 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:07:10 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 12 files
16/04/11 14:07:10 INFO fs.TestDFSIO: created control files for: 12 files
16/04/11 14:07:11 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:07:11 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:07:11 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:07:11 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:07:11 INFO mapred.FileInputFormat: Total input paths to process : 12
16/04/11 14:07:11 INFO mapreduce.JobSubmitter: number of splits:12
16/04/11 14:07:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0014
16/04/11 14:07:12 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0014
16/04/11 14:07:12 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0014/
16/04/11 14:07:12 INFO mapreduce.Job: Running job: job_1460382022143_0014
16/04/11 14:07:18 INFO mapreduce.Job: Job job_1460382022143_0014 running in uber mode : false
16/04/11 14:07:18 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:07:29 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:08:42 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 14:08:44 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 14:08:48 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 14:08:49 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 14:08:50 INFO mapreduce.Job: map 86% reduce 0%
16/04/11 14:08:51 INFO mapreduce.Job: map 89% reduce 0%
16/04/11 14:08:52 INFO mapreduce.Job: map 89% reduce 22%
16/04/11 14:08:55 INFO mapreduce.Job: map 92% reduce 22%
16/04/11 14:08:56 INFO mapreduce.Job: map 97% reduce 22%
16/04/11 14:08:57 INFO mapreduce.Job: map 100% reduce 22%
16/04/11 14:08:58 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:08:58 INFO mapreduce.Job: Job job_1460382022143_0014 completed successfully
16/04/11 14:08:59 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1047
FILE: Number of bytes written=1736209
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3076
HDFS: Number of bytes written=103079215188
HDFS: Number of read operations=51
HDFS: Number of large read operations=0
HDFS: Number of write operations=14
Job Counters
Launched map tasks=12
Launched reduce tasks=1
Data-local map tasks=12
Total time spent by all maps in occupied slots (ms)=1080993
Total time spent by all reduces in occupied slots (ms)=27400
Total time spent by all map tasks (ms)=1080993
Total time spent by all reduce tasks (ms)=13700
Total vcore-seconds taken by all map tasks=1080993
Total vcore-seconds taken by all reduce tasks=13700
Total megabyte-seconds taken by all map tasks=3874278912
Total megabyte-seconds taken by all reduce tasks=98201600
Map-Reduce Framework
Map input records=12
Map output records=60
Map output bytes=921
Map output materialized bytes=1113
Input split bytes=1730
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1113
Reduce input records=60
Reduce output records=5
Spilled Records=120
Shuffled Maps =12
Failed Shuffles=0
Merged Map outputs=12
GC time elapsed (ms)=9235
CPU time spent (ms)=431910
Physical memory (bytes) snapshot=30614167552
Virtual memory (bytes) snapshot=51159031808
Total committed heap usage (bytes)=37693685760
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1346
File Output Format Counters
Bytes Written=84
16/04/11 14:08:59 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:08:59 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:08:59 UTC 2016
16/04/11 14:08:59 INFO fs.TestDFSIO: Number of files: 12
16/04/11 14:08:59 INFO fs.TestDFSIO: Total MBytes processed: 98304.0
16/04/11 14:08:59 INFO fs.TestDFSIO: Throughput mb/sec: 94.88922158289631
16/04/11 14:08:59 INFO fs.TestDFSIO: Average IO rate mb/sec: 95.18067169189453
16/04/11 14:08:59 INFO fs.TestDFSIO: IO rate std deviation: 5.294647388380744
16/04/11 14:08:59 INFO fs.TestDFSIO: Test exec time sec: 108.318
16/04/11 14:08:59 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:09:00 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:09:00 INFO fs.TestDFSIO: nrFiles = 16
16/04/11 14:09:00 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:09:00 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:09:00 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:09:01 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 16 files
16/04/11 14:09:01 INFO fs.TestDFSIO: created control files for: 16 files
16/04/11 14:09:02 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:09:02 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:09:02 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:09:02 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:09:03 INFO mapred.FileInputFormat: Total input paths to process : 16
16/04/11 14:09:03 INFO mapreduce.JobSubmitter: number of splits:16
16/04/11 14:09:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0015
16/04/11 14:09:03 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0015
16/04/11 14:09:03 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0015/
16/04/11 14:09:03 INFO mapreduce.Job: Running job: job_1460382022143_0015
16/04/11 14:09:10 INFO mapreduce.Job: Job job_1460382022143_0015 running in uber mode : false
16/04/11 14:09:10 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:09:20 INFO mapreduce.Job: map 33% reduce 0%
16/04/11 14:09:21 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:11:33 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 14:11:35 INFO mapreduce.Job: map 73% reduce 0%
16/04/11 14:11:39 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 14:11:41 INFO mapreduce.Job: map 77% reduce 0%
16/04/11 14:11:42 INFO mapreduce.Job: map 79% reduce 0%
16/04/11 14:11:43 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 14:11:44 INFO mapreduce.Job: map 85% reduce 13%
16/04/11 14:11:45 INFO mapreduce.Job: map 94% reduce 13%
16/04/11 14:11:46 INFO mapreduce.Job: map 96% reduce 13%
16/04/11 14:11:47 INFO mapreduce.Job: map 96% reduce 29%
16/04/11 14:11:48 INFO mapreduce.Job: map 100% reduce 29%
16/04/11 14:11:49 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:11:49 INFO mapreduce.Job: Job job_1460382022143_0015 completed successfully
16/04/11 14:11:49 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=1407
FILE: Number of bytes written=2270519
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=4108
HDFS: Number of bytes written=137438953555
HDFS: Number of read operations=67
HDFS: Number of large read operations=0
HDFS: Number of write operations=18
Job Counters
Launched map tasks=16
Launched reduce tasks=1
Data-local map tasks=13
Rack-local map tasks=3
Total time spent by all maps in occupied slots (ms)=2403882
Total time spent by all reduces in occupied slots (ms)=27930
Total time spent by all map tasks (ms)=2403882
Total time spent by all reduce tasks (ms)=13965
Total vcore-seconds taken by all map tasks=2403882
Total vcore-seconds taken by all reduce tasks=13965
Total megabyte-seconds taken by all map tasks=8615513088
Total megabyte-seconds taken by all reduce tasks=100101120
Map-Reduce Framework
Map input records=16
Map output records=80
Map output bytes=1241
Map output materialized bytes=1497
Input split bytes=2310
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1497
Reduce input records=80
Reduce output records=5
Spilled Records=160
Shuffled Maps =16
Failed Shuffles=0
Merged Map outputs=16
GC time elapsed (ms)=13717
CPU time spent (ms)=594950
Physical memory (bytes) snapshot=40246849536
Virtual memory (bytes) snapshot=65938083840
Total committed heap usage (bytes)=48923410432
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1798
File Output Format Counters
Bytes Written=83
16/04/11 14:11:49 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:11:49 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:11:49 UTC 2016
16/04/11 14:11:49 INFO fs.TestDFSIO: Number of files: 16
16/04/11 14:11:49 INFO fs.TestDFSIO: Total MBytes processed: 131072.0
16/04/11 14:11:49 INFO fs.TestDFSIO: Throughput mb/sec: 56.042127370648004
16/04/11 14:11:49 INFO fs.TestDFSIO: Average IO rate mb/sec: 56.09883117675781
16/04/11 14:11:49 INFO fs.TestDFSIO: IO rate std deviation: 1.8046023995261853
16/04/11 14:11:49 INFO fs.TestDFSIO: Test exec time sec: 167.548
16/04/11 14:11:49 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:11:50 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:11:50 INFO fs.TestDFSIO: nrFiles = 20
16/04/11 14:11:50 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:11:50 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:11:50 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:11:51 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 20 files
16/04/11 14:11:52 INFO fs.TestDFSIO: created control files for: 20 files
16/04/11 14:11:52 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:11:52 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:11:53 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:11:53 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:11:53 INFO mapred.FileInputFormat: Total input paths to process : 20
16/04/11 14:11:53 INFO mapreduce.JobSubmitter: number of splits:20
16/04/11 14:11:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0016
16/04/11 14:11:53 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0016
16/04/11 14:11:53 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0016/
16/04/11 14:11:53 INFO mapreduce.Job: Running job: job_1460382022143_0016
16/04/11 14:11:59 INFO mapreduce.Job: Job job_1460382022143_0016 running in uber mode : false
16/04/11 14:11:59 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:12:11 INFO mapreduce.Job: map 40% reduce 0%
16/04/11 14:12:12 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:14:38 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 14:14:44 INFO mapreduce.Job: map 70% reduce 0%
16/04/11 14:14:49 INFO mapreduce.Job: map 72% reduce 3%
16/04/11 14:14:52 INFO mapreduce.Job: map 72% reduce 5%
16/04/11 14:14:58 INFO mapreduce.Job: map 73% reduce 5%
16/04/11 14:15:01 INFO mapreduce.Job: map 75% reduce 7%
16/04/11 14:15:04 INFO mapreduce.Job: map 77% reduce 8%
16/04/11 14:15:05 INFO mapreduce.Job: map 78% reduce 8%
16/04/11 14:15:07 INFO mapreduce.Job: map 80% reduce 12%
16/04/11 14:15:08 INFO mapreduce.Job: map 82% reduce 12%
16/04/11 14:15:10 INFO mapreduce.Job: map 83% reduce 15%
16/04/11 14:15:12 INFO mapreduce.Job: map 85% reduce 15%
16/04/11 14:15:13 INFO mapreduce.Job: map 85% reduce 18%
16/04/11 14:15:14 INFO mapreduce.Job: map 90% reduce 18%
16/04/11 14:15:15 INFO mapreduce.Job: map 93% reduce 18%
16/04/11 14:15:16 INFO mapreduce.Job: map 98% reduce 25%
16/04/11 14:15:17 INFO mapreduce.Job: map 100% reduce 25%
16/04/11 14:15:18 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:15:19 INFO mapreduce.Job: Job job_1460382022143_0016 completed successfully
16/04/11 14:15:20 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1756
FILE: Number of bytes written=2804831
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5140
HDFS: Number of bytes written=171798691923
HDFS: Number of read operations=83
HDFS: Number of large read operations=0
HDFS: Number of write operations=22
Job Counters
Launched map tasks=20
Launched reduce tasks=1
Data-local map tasks=20
Total time spent by all maps in occupied slots (ms)=3701871
Total time spent by all reduces in occupied slots (ms)=75496
Total time spent by all map tasks (ms)=3701871
Total time spent by all reduce tasks (ms)=37748
Total vcore-seconds taken by all map tasks=3701871
Total vcore-seconds taken by all reduce tasks=37748
Total megabyte-seconds taken by all map tasks=13267505664
Total megabyte-seconds taken by all reduce tasks=270577664
Map-Reduce Framework
Map input records=20
Map output records=100
Map output bytes=1550
Map output materialized bytes=1870
Input split bytes=2890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1870
Reduce input records=100
Reduce output records=5
Spilled Records=200
Shuffled Maps =20
Failed Shuffles=0
Merged Map outputs=20
GC time elapsed (ms)=18898
CPU time spent (ms)=784010
Physical memory (bytes) snapshot=50685235200
Virtual memory (bytes) snapshot=80719581184
Total committed heap usage (bytes)=60722511872
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2250
File Output Format Counters
Bytes Written=83
16/04/11 14:15:20 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:15:20 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:15:20 UTC 2016
16/04/11 14:15:20 INFO fs.TestDFSIO: Number of files: 20
16/04/11 14:15:20 INFO fs.TestDFSIO: Total MBytes processed: 163840.0
16/04/11 14:15:20 INFO fs.TestDFSIO: Throughput mb/sec: 45.22539964882963
16/04/11 14:15:20 INFO fs.TestDFSIO: Average IO rate mb/sec: 45.41441345214844
16/04/11 14:15:20 INFO fs.TestDFSIO: IO rate std deviation: 3.0638224211871417
16/04/11 14:15:20 INFO fs.TestDFSIO: Test exec time sec: 207.733
16/04/11 14:15:20 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:15:21 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:15:21 INFO fs.TestDFSIO: nrFiles = 24
16/04/11 14:15:21 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:15:21 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:15:21 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:15:22 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 24 files
16/04/11 14:15:23 INFO fs.TestDFSIO: created control files for: 24 files
16/04/11 14:15:23 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:15:23 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:15:23 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:15:23 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:15:24 INFO mapred.FileInputFormat: Total input paths to process : 24
16/04/11 14:15:24 INFO mapreduce.JobSubmitter: number of splits:24
16/04/11 14:15:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0017
16/04/11 14:15:24 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0017
16/04/11 14:15:24 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0017/
16/04/11 14:15:24 INFO mapreduce.Job: Running job: job_1460382022143_0017
16/04/11 14:15:30 INFO mapreduce.Job: Job job_1460382022143_0017 running in uber mode : false
16/04/11 14:15:30 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:15:41 INFO mapreduce.Job: map 36% reduce 0%
16/04/11 14:15:42 INFO mapreduce.Job: map 42% reduce 0%
16/04/11 14:15:43 INFO mapreduce.Job: map 64% reduce 0%
16/04/11 14:15:44 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:18:13 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 14:18:14 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 14:18:23 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 14:18:24 INFO mapreduce.Job: map 72% reduce 6%
16/04/11 14:18:25 INFO mapreduce.Job: map 74% reduce 6%
16/04/11 14:18:26 INFO mapreduce.Job: map 75% reduce 6%
16/04/11 14:18:27 INFO mapreduce.Job: map 76% reduce 8%
16/04/11 14:18:29 INFO mapreduce.Job: map 78% reduce 8%
16/04/11 14:18:30 INFO mapreduce.Job: map 81% reduce 10%
16/04/11 14:18:31 INFO mapreduce.Job: map 83% reduce 10%
16/04/11 14:18:33 INFO mapreduce.Job: map 83% reduce 17%
16/04/11 14:18:35 INFO mapreduce.Job: map 85% reduce 17%
16/04/11 14:18:36 INFO mapreduce.Job: map 85% reduce 18%
16/04/11 14:18:38 INFO mapreduce.Job: map 86% reduce 18%
16/04/11 14:18:39 INFO mapreduce.Job: map 86% reduce 19%
16/04/11 14:18:40 INFO mapreduce.Job: map 89% reduce 19%
16/04/11 14:18:42 INFO mapreduce.Job: map 89% reduce 22%
16/04/11 14:18:47 INFO mapreduce.Job: map 90% reduce 22%
16/04/11 14:18:48 INFO mapreduce.Job: map 90% reduce 24%
16/04/11 14:18:49 INFO mapreduce.Job: map 92% reduce 24%
16/04/11 14:18:50 INFO mapreduce.Job: map 93% reduce 24%
16/04/11 14:18:51 INFO mapreduce.Job: map 94% reduce 26%
16/04/11 14:18:53 INFO mapreduce.Job: map 99% reduce 26%
16/04/11 14:18:54 INFO mapreduce.Job: map 99% reduce 32%
16/04/11 14:18:55 INFO mapreduce.Job: map 100% reduce 32%
16/04/11 14:18:56 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:18:56 INFO mapreduce.Job: Job job_1460382022143_0017 completed successfully
16/04/11 14:18:56 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2109
FILE: Number of bytes written=3339125
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=6172
HDFS: Number of bytes written=206158430292
HDFS: Number of read operations=99
HDFS: Number of large read operations=0
HDFS: Number of write operations=26
Job Counters
Launched map tasks=24
Launched reduce tasks=1
Data-local map tasks=24
Total time spent by all maps in occupied slots (ms)=4406119
Total time spent by all reduces in occupied slots (ms)=80244
Total time spent by all map tasks (ms)=4406119
Total time spent by all reduce tasks (ms)=40122
Total vcore-seconds taken by all map tasks=4406119
Total vcore-seconds taken by all reduce tasks=40122
Total megabyte-seconds taken by all map tasks=15791530496
Total megabyte-seconds taken by all reduce tasks=287594496
Map-Reduce Framework
Map input records=24
Map output records=120
Map output bytes=1863
Map output materialized bytes=2247
Input split bytes=3470
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2247
Reduce input records=120
Reduce output records=5
Spilled Records=240
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=25501
CPU time spent (ms)=1008590
Physical memory (bytes) snapshot=59677818880
Virtual memory (bytes) snapshot=95483994112
Total committed heap usage (bytes)=71717355520
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2702
File Output Format Counters
Bytes Written=84
16/04/11 14:18:56 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:18:56 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:18:56 UTC 2016
16/04/11 14:18:56 INFO fs.TestDFSIO: Number of files: 24
16/04/11 14:18:56 INFO fs.TestDFSIO: Total MBytes processed: 196608.0
16/04/11 14:18:56 INFO fs.TestDFSIO: Throughput mb/sec: 45.829680433737195
16/04/11 14:18:56 INFO fs.TestDFSIO: Average IO rate mb/sec: 46.02585983276367
16/04/11 14:18:56 INFO fs.TestDFSIO: IO rate std deviation: 3.02800439880239
16/04/11 14:18:56 INFO fs.TestDFSIO: Test exec time sec: 213.82
16/04/11 14:18:56 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:18:58 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:18:58 INFO fs.TestDFSIO: nrFiles = 28
16/04/11 14:18:58 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:18:58 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:18:58 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:18:59 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 28 files
16/04/11 14:18:59 INFO fs.TestDFSIO: created control files for: 28 files
16/04/11 14:19:00 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:19:00 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:19:00 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:19:00 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:19:00 INFO mapred.FileInputFormat: Total input paths to process : 28
16/04/11 14:19:01 INFO mapreduce.JobSubmitter: number of splits:28
16/04/11 14:19:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0018
16/04/11 14:19:01 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0018
16/04/11 14:19:01 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0018/
16/04/11 14:19:01 INFO mapreduce.Job: Running job: job_1460382022143_0018
16/04/11 14:19:08 INFO mapreduce.Job: Job job_1460382022143_0018 running in uber mode : false
16/04/11 14:19:08 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:19:18 INFO mapreduce.Job: map 2% reduce 0%
16/04/11 14:19:19 INFO mapreduce.Job: map 24% reduce 0%
16/04/11 14:19:20 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:22:02 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 14:22:05 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 14:22:12 INFO mapreduce.Job: map 73% reduce 0%
16/04/11 14:22:16 INFO mapreduce.Job: map 73% reduce 6%
16/04/11 14:22:17 INFO mapreduce.Job: map 74% reduce 6%
16/04/11 14:22:19 INFO mapreduce.Job: map 76% reduce 7%
16/04/11 14:22:21 INFO mapreduce.Job: map 77% reduce 7%
16/04/11 14:22:22 INFO mapreduce.Job: map 80% reduce 10%
16/04/11 14:22:23 INFO mapreduce.Job: map 81% reduce 10%
16/04/11 14:22:24 INFO mapreduce.Job: map 82% reduce 10%
16/04/11 14:22:25 INFO mapreduce.Job: map 82% reduce 14%
16/04/11 14:22:28 INFO mapreduce.Job: map 82% reduce 15%
16/04/11 14:22:29 INFO mapreduce.Job: map 83% reduce 15%
16/04/11 14:22:30 INFO mapreduce.Job: map 86% reduce 15%
16/04/11 14:22:31 INFO mapreduce.Job: map 86% reduce 17%
16/04/11 14:22:34 INFO mapreduce.Job: map 87% reduce 19%
16/04/11 14:22:36 INFO mapreduce.Job: map 88% reduce 19%
16/04/11 14:22:37 INFO mapreduce.Job: map 88% reduce 21%
16/04/11 14:22:43 INFO mapreduce.Job: map 89% reduce 21%
16/04/11 14:22:44 INFO mapreduce.Job: map 90% reduce 21%
16/04/11 14:22:46 INFO mapreduce.Job: map 90% reduce 24%
16/04/11 14:22:47 INFO mapreduce.Job: map 93% reduce 24%
16/04/11 14:22:48 INFO mapreduce.Job: map 94% reduce 24%
16/04/11 14:22:49 INFO mapreduce.Job: map 94% reduce 26%
16/04/11 14:22:51 INFO mapreduce.Job: map 95% reduce 26%
16/04/11 14:22:52 INFO mapreduce.Job: map 100% reduce 27%
16/04/11 14:22:53 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:22:53 INFO mapreduce.Job: Job job_1460382022143_0018 completed successfully
16/04/11 14:22:53 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=2463
FILE: Number of bytes written=3873413
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=7204
HDFS: Number of bytes written=240518168660
HDFS: Number of read operations=115
HDFS: Number of large read operations=0
HDFS: Number of write operations=30
Job Counters
Launched map tasks=28
Launched reduce tasks=1
Data-local map tasks=27
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5618374
Total time spent by all reduces in occupied slots (ms)=91932
Total time spent by all map tasks (ms)=5618374
Total time spent by all reduce tasks (ms)=45966
Total vcore-seconds taken by all map tasks=5618374
Total vcore-seconds taken by all reduce tasks=45966
Total megabyte-seconds taken by all map tasks=20136252416
Total megabyte-seconds taken by all reduce tasks=329484288
Map-Reduce Framework
Map input records=28
Map output records=140
Map output bytes=2177
Map output materialized bytes=2625
Input split bytes=4050
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2625
Reduce input records=140
Reduce output records=5
Spilled Records=280
Shuffled Maps =28
Failed Shuffles=0
Merged Map outputs=28
GC time elapsed (ms)=29388
CPU time spent (ms)=1169190
Physical memory (bytes) snapshot=69481463808
Virtual memory (bytes) snapshot=110254178304
Total committed heap usage (bytes)=82978537472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=3154
File Output Format Counters
Bytes Written=84
16/04/11 14:22:53 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:22:53 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:22:53 UTC 2016
16/04/11 14:22:53 INFO fs.TestDFSIO: Number of files: 28
16/04/11 14:22:53 INFO fs.TestDFSIO: Total MBytes processed: 229376.0
16/04/11 14:22:53 INFO fs.TestDFSIO: Throughput mb/sec: 41.85189375002281
16/04/11 14:22:53 INFO fs.TestDFSIO: Average IO rate mb/sec: 42.11272048950195
16/04/11 14:22:53 INFO fs.TestDFSIO: IO rate std deviation: 3.33967311690379
16/04/11 14:22:53 INFO fs.TestDFSIO: Test exec time sec: 233.883
16/04/11 14:22:53 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:22:55 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:22:55 INFO fs.TestDFSIO: nrFiles = 32
16/04/11 14:22:55 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:22:55 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:22:55 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:22:56 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 32 files
16/04/11 14:22:56 INFO fs.TestDFSIO: created control files for: 32 files
16/04/11 14:22:57 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:22:57 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:22:57 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:22:57 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:22:57 INFO mapred.FileInputFormat: Total input paths to process : 32
16/04/11 14:22:57 INFO mapreduce.JobSubmitter: number of splits:32
16/04/11 14:22:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0019
16/04/11 14:22:58 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0019
16/04/11 14:22:58 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0019/
16/04/11 14:22:58 INFO mapreduce.Job: Running job: job_1460382022143_0019
16/04/11 14:23:05 INFO mapreduce.Job: Job job_1460382022143_0019 running in uber mode : false
16/04/11 14:23:05 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:23:16 INFO mapreduce.Job: map 17% reduce 0%
16/04/11 14:23:17 INFO mapreduce.Job: map 33% reduce 0%
16/04/11 14:23:18 INFO mapreduce.Job: map 54% reduce 0%
16/04/11 14:23:19 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:26:15 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 14:26:29 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 14:26:31 INFO mapreduce.Job: map 70% reduce 0%
16/04/11 14:26:40 INFO mapreduce.Job: map 71% reduce 3%
16/04/11 14:26:42 INFO mapreduce.Job: map 72% reduce 3%
16/04/11 14:26:43 INFO mapreduce.Job: map 73% reduce 4%
16/04/11 14:26:45 INFO mapreduce.Job: map 74% reduce 4%
16/04/11 14:26:46 INFO mapreduce.Job: map 75% reduce 6%
16/04/11 14:26:47 INFO mapreduce.Job: map 76% reduce 6%
16/04/11 14:26:48 INFO mapreduce.Job: map 77% reduce 6%
16/04/11 14:26:49 INFO mapreduce.Job: map 78% reduce 10%
16/04/11 14:26:50 INFO mapreduce.Job: map 79% reduce 10%
16/04/11 14:26:51 INFO mapreduce.Job: map 81% reduce 10%
16/04/11 14:26:52 INFO mapreduce.Job: map 82% reduce 15%
16/04/11 14:26:55 INFO mapreduce.Job: map 83% reduce 16%
16/04/11 14:26:56 INFO mapreduce.Job: map 84% reduce 16%
16/04/11 14:26:57 INFO mapreduce.Job: map 85% reduce 16%
16/04/11 14:26:58 INFO mapreduce.Job: map 85% reduce 19%
16/04/11 14:27:02 INFO mapreduce.Job: map 86% reduce 19%
16/04/11 14:27:04 INFO mapreduce.Job: map 88% reduce 20%
16/04/11 14:27:05 INFO mapreduce.Job: map 90% reduce 20%
16/04/11 14:27:06 INFO mapreduce.Job: map 91% reduce 20%
16/04/11 14:27:07 INFO mapreduce.Job: map 91% reduce 24%
16/04/11 14:27:09 INFO mapreduce.Job: map 92% reduce 24%
16/04/11 14:27:10 INFO mapreduce.Job: map 92% reduce 25%
16/04/11 14:27:12 INFO mapreduce.Job: map 93% reduce 25%
16/04/11 14:27:13 INFO mapreduce.Job: map 94% reduce 26%
16/04/11 14:27:14 INFO mapreduce.Job: map 95% reduce 26%
16/04/11 14:27:15 INFO mapreduce.Job: map 96% reduce 26%
16/04/11 14:27:16 INFO mapreduce.Job: map 97% reduce 29%
16/04/11 14:27:17 INFO mapreduce.Job: map 99% reduce 29%
16/04/11 14:27:18 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:27:19 INFO mapreduce.Job: Job job_1460382022143_0019 completed successfully
16/04/11 14:27:19 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2808
FILE: Number of bytes written=4407716
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=8236
HDFS: Number of bytes written=274877907028
HDFS: Number of read operations=131
HDFS: Number of large read operations=0
HDFS: Number of write operations=34
Job Counters
Launched map tasks=32
Launched reduce tasks=1
Data-local map tasks=32
Total time spent by all maps in occupied slots (ms)=7325577
Total time spent by all reduces in occupied slots (ms)=94056
Total time spent by all map tasks (ms)=7325577
Total time spent by all reduce tasks (ms)=47028
Total vcore-seconds taken by all map tasks=7325577
Total vcore-seconds taken by all reduce tasks=47028
Total megabyte-seconds taken by all map tasks=26254867968
Total megabyte-seconds taken by all reduce tasks=337096704
Map-Reduce Framework
Map input records=32
Map output records=160
Map output bytes=2482
Map output materialized bytes=2994
Input split bytes=4630
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2994
Reduce input records=160
Reduce output records=5
Spilled Records=320
Shuffled Maps =32
Failed Shuffles=0
Merged Map outputs=32
GC time elapsed (ms)=35003
CPU time spent (ms)=1354970
Physical memory (bytes) snapshot=78567866368
Virtual memory (bytes) snapshot=124959154176
Total committed heap usage (bytes)=94276419584
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=3606
File Output Format Counters
Bytes Written=84
16/04/11 14:27:19 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:27:19 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:27:19 UTC 2016
16/04/11 14:27:19 INFO fs.TestDFSIO: Number of files: 32
16/04/11 14:27:19 INFO fs.TestDFSIO: Total MBytes processed: 262144.0
16/04/11 14:27:19 INFO fs.TestDFSIO: Throughput mb/sec: 36.638218694710346
16/04/11 14:27:19 INFO fs.TestDFSIO: Average IO rate mb/sec: 36.8348503112793
16/04/11 14:27:19 INFO fs.TestDFSIO: IO rate std deviation: 2.7498142349797896
16/04/11 14:27:19 INFO fs.TestDFSIO: Test exec time sec: 263.01
16/04/11 14:27:19 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 14:27:21 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 14:27:21 INFO fs.TestDFSIO: nrFiles = 36
16/04/11 14:27:21 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 14:27:21 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 14:27:21 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 14:27:22 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 36 files
16/04/11 14:27:22 INFO fs.TestDFSIO: created control files for: 36 files
16/04/11 14:27:23 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:27:23 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:27:23 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 14:27:23 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 14:27:24 INFO mapred.FileInputFormat: Total input paths to process : 36
16/04/11 14:27:24 INFO mapreduce.JobSubmitter: number of splits:36
16/04/11 14:27:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0020
16/04/11 14:27:24 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0020
16/04/11 14:27:24 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0020/
16/04/11 14:27:24 INFO mapreduce.Job: Running job: job_1460382022143_0020
16/04/11 14:27:31 INFO mapreduce.Job: Job job_1460382022143_0020 running in uber mode : false
16/04/11 14:27:31 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 14:27:41 INFO mapreduce.Job: map 11% reduce 0%
16/04/11 14:27:42 INFO mapreduce.Job: map 35% reduce 0%
16/04/11 14:27:43 INFO mapreduce.Job: map 48% reduce 0%
16/04/11 14:27:45 INFO mapreduce.Job: map 65% reduce 0%
16/04/11 14:30:55 INFO mapreduce.Job: map 66% reduce 0%
16/04/11 14:31:01 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 14:31:05 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 14:31:13 INFO mapreduce.Job: map 70% reduce 3%
16/04/11 14:31:16 INFO mapreduce.Job: map 71% reduce 4%
16/04/11 14:31:19 INFO mapreduce.Job: map 71% reduce 5%
16/04/11 14:31:22 INFO mapreduce.Job: map 73% reduce 5%
16/04/11 14:31:24 INFO mapreduce.Job: map 75% reduce 5%
16/04/11 14:31:25 INFO mapreduce.Job: map 75% reduce 6%
16/04/11 14:31:26 INFO mapreduce.Job: map 76% reduce 6%
16/04/11 14:31:28 INFO mapreduce.Job: map 76% reduce 9%
16/04/11 14:31:31 INFO mapreduce.Job: map 77% reduce 9%
16/04/11 14:31:34 INFO mapreduce.Job: map 79% reduce 10%
16/04/11 14:31:35 INFO mapreduce.Job: map 80% reduce 10%
16/04/11 14:31:37 INFO mapreduce.Job: map 81% reduce 13%
16/04/11 14:31:39 INFO mapreduce.Job: map 84% reduce 13%
16/04/11 14:31:40 INFO mapreduce.Job: map 86% reduce 16%
16/04/11 14:31:41 INFO mapreduce.Job: map 87% reduce 16%
16/04/11 14:31:43 INFO mapreduce.Job: map 87% reduce 20%
16/04/11 14:31:45 INFO mapreduce.Job: map 88% reduce 20%
16/04/11 14:31:46 INFO mapreduce.Job: map 89% reduce 20%
16/04/11 14:31:47 INFO mapreduce.Job: map 90% reduce 20%
16/04/11 14:31:49 INFO mapreduce.Job: map 90% reduce 23%
16/04/11 14:31:50 INFO mapreduce.Job: map 91% reduce 23%
16/04/11 14:31:51 INFO mapreduce.Job: map 92% reduce 23%
16/04/11 14:31:52 INFO mapreduce.Job: map 92% reduce 24%
16/04/11 14:31:55 INFO mapreduce.Job: map 92% reduce 25%
16/04/11 14:31:56 INFO mapreduce.Job: map 93% reduce 25%
16/04/11 14:31:57 INFO mapreduce.Job: map 94% reduce 25%
16/04/11 14:31:58 INFO mapreduce.Job: map 94% reduce 27%
16/04/11 14:32:00 INFO mapreduce.Job: map 95% reduce 27%
16/04/11 14:32:01 INFO mapreduce.Job: map 95% reduce 28%
16/04/11 14:32:03 INFO mapreduce.Job: map 97% reduce 28%
16/04/11 14:32:04 INFO mapreduce.Job: map 98% reduce 29%
16/04/11 14:32:06 INFO mapreduce.Job: map 99% reduce 29%
16/04/11 14:32:07 INFO mapreduce.Job: map 99% reduce 31%
16/04/11 14:32:10 INFO mapreduce.Job: map 99% reduce 32%
16/04/11 14:32:17 INFO mapreduce.Job: map 100% reduce 32%
16/04/11 14:32:18 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:32:18 INFO mapreduce.Job: Job job_1460382022143_0020 completed successfully
16/04/11 14:32:19 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=3161
FILE: Number of bytes written=4942012
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=9268
HDFS: Number of bytes written=309237645396
HDFS: Number of read operations=147
HDFS: Number of large read operations=0
HDFS: Number of write operations=38
Job Counters
Launched map tasks=36
Launched reduce tasks=1
Data-local map tasks=36
Total time spent by all maps in occupied slots (ms)=8660657
Total time spent by all reduces in occupied slots (ms)=147892
Total time spent by all map tasks (ms)=8660657
Total time spent by all reduce tasks (ms)=73946
Total vcore-seconds taken by all map tasks=8660657
Total vcore-seconds taken by all reduce tasks=73946
Total megabyte-seconds taken by all map tasks=31039794688
Total megabyte-seconds taken by all reduce tasks=530044928
Map-Reduce Framework
Map input records=36
Map output records=180
Map output bytes=2795
Map output materialized bytes=3371
Input split bytes=5210
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=3371
Reduce input records=180
Reduce output records=5
Spilled Records=360
Shuffled Maps =36
Failed Shuffles=0
Merged Map outputs=36
GC time elapsed (ms)=40572
CPU time spent (ms)=1549750
Physical memory (bytes) snapshot=88700796928
Virtual memory (bytes) snapshot=139687620608
Total committed heap usage (bytes)=105862135808
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=4058
File Output Format Counters
Bytes Written=84
16/04/11 14:32:19 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:32:19 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:32:19 UTC 2016
16/04/11 14:32:19 INFO fs.TestDFSIO: Number of files: 36
16/04/11 14:32:19 INFO fs.TestDFSIO: Total MBytes processed: 294912.0
16/04/11 14:32:19 INFO fs.TestDFSIO: Throughput mb/sec: 34.85456463940881
16/04/11 14:32:19 INFO fs.TestDFSIO: Average IO rate mb/sec: 36.3818473815918
16/04/11 14:32:19 INFO fs.TestDFSIO: IO rate std deviation: 11.999556447247837
16/04/11 14:32:19 INFO fs.TestDFSIO: Test exec time sec: 296.131
16/04/11 14:32:19 INFO fs.TestDFSIO:
[hdfs@docker-ambari ~]#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment