第1章 數(shù)據(jù)中心概述 ······································································.1
1.1 數(shù)據(jù)中心簡(jiǎn)介 ····················································································.2
1.1.1 什么是數(shù)據(jù)中心 ·········································································.2
1.1.2 數(shù)據(jù)中心的需求和挑戰(zhàn) ································································.4
1.2 云計(jì)算數(shù)據(jù)中心資源調(diào)度需求分析 ·························································.4
1.2.1 技術(shù)需求 ··················································································.4
1.2.2 技術(shù)目標(biāo) ··················································································.5
1.3 云計(jì)算數(shù)據(jù)中心資源調(diào)度研究進(jìn)展 ·························································.5
1.4 云計(jì)算數(shù)據(jù)中心資源調(diào)度方案分析 ·························································.6
1.4.1 Google解決方案 ········································································.6
1.4.2 Amazon解決方案 ·······································································.7
1.4.3 IBM解決方案 ············································································.8
1.4.4 HP解決方案··············································································10
1.4.5 VMware解決方案 ·······································································10
1.4.6 阿里云解決方案 ·········································································12
1.4.7 華為云解決方案 ·········································································14
1.4.8 其他廠(chǎng)家解決方案 ······································································15
1.5 云計(jì)算數(shù)據(jù)中心資源調(diào)度標(biāo)準(zhǔn)進(jìn)展 ·························································17
1.6 云資源管理調(diào)度關(guān)鍵技術(shù)及研究熱點(diǎn) ······················································18
本章小結(jié) ·································································································20
思考題·····································································································21
參考文獻(xiàn) ·································································································21
第2章 云計(jì)算概述 ········································································.25
2.1 云計(jì)算的發(fā)展背景 ··············································································26
2.2 云計(jì)算是集大成者 ··············································································28
2.2.1 并行計(jì)算 ················································································.28
2.2.2 網(wǎng)格計(jì)算 ················································································.29
2.2.3 效用計(jì)算 ················································································.29
2.2.4 普適計(jì)算 ················································································.30
2.2.5 軟件即服務(wù) ·············································································.30
2.2.6 虛擬化技術(shù) ·············································································.31
2.3 云計(jì)算的驅(qū)動(dòng)因素 ············································································.31
2.3.1 云計(jì)算的發(fā)展現(xiàn)狀和趨勢(shì) ···························································.33
2.3.2 云計(jì)算應(yīng)用初步分類(lèi) ·································································.35
2.4 云計(jì)算產(chǎn)業(yè)鏈中的不同角色 ································································.36
2.5 云計(jì)算的主要特征和技術(shù)挑戰(zhàn) ·····························································.37
2.5.1 云計(jì)算的主要特征 ····································································.37
2.5.2 挑戰(zhàn)性問(wèn)題 ·············································································.38
本章小結(jié) ································································································.46
思考題 ···································································································.46
參考文獻(xiàn) ································································································.46
第3章 大數(shù)據(jù)處理 ········································································.51
3.1 大數(shù)據(jù)的發(fā)展背景及定義 ···································································.52
3.2 大數(shù)據(jù)問(wèn)題 ······················································································.55
3.2.1 速度方面的問(wèn)題 ·······································································.55
3.2.2 種類(lèi)及架構(gòu)問(wèn)題 ·······································································.56
3.2.3 體量及靈活性問(wèn)題 ····································································.56
3.2.4 成本問(wèn)題 ················································································.57
3.2.5 價(jià)值挖掘問(wèn)題 ··········································································.57
3.2.6 存儲(chǔ)及安全問(wèn)題 ·······································································.58
3.2.7 互聯(lián)互通與數(shù)據(jù)共享問(wèn)題 ···························································.59
3.3 大數(shù)據(jù)與云計(jì)算的辯證關(guān)系 ································································.60
3.4 大數(shù)據(jù)技術(shù) ······················································································.61
3.4.1 基礎(chǔ)架構(gòu)支持 ··········································································.62
3.4.2 數(shù)據(jù)采集 ················································································.64
3.4.3 數(shù)據(jù)存儲(chǔ) ················································································.65
3.4.4 數(shù)據(jù)計(jì)算 ················································································.68
3.4.5 數(shù)據(jù)展現(xiàn)與交互 ·······································································.73
本章小結(jié) ································································································.75
思考題 ···································································································.76
參考文獻(xiàn) ································································································.76
第4章 人工智能平臺(tái)的資源調(diào)度概述 ················································.78
4.1 引言 ·································································································79
4.2 深度學(xué)習(xí)的分布式并行訓(xùn)練系統(tǒng)架構(gòu) ······················································79
4.3 深度學(xué)習(xí)的分布式并行策略 ··································································81
4.3.1 深度學(xué)習(xí)的基礎(chǔ)概念 ···································································82
4.3.2 分布式并行訓(xùn)練算法 ···································································82
4.3.3 研究現(xiàn)狀分析 ············································································85
4.4 分布式并行訓(xùn)練的時(shí)效分析 ··································································91
本章小結(jié) ·································································································96
思考題·····································································································96
參考文獻(xiàn) ·································································································97
第5章 基于深度學(xué)習(xí)的云服務(wù)負(fù)載預(yù)測(cè)方法 ·······································.99
5.1 引言 ······························································································.100
5.2 相關(guān)工作 ························································································.101
5.2.1 基于回歸方法的云服務(wù)負(fù)載預(yù)測(cè)方法···········································.101
5.2.2 基于學(xué)習(xí)的云服務(wù)負(fù)載預(yù)測(cè)方法 ·················································.102
5.2.3 討論分析 ···············································································.103
5.3 系統(tǒng)模型 ························································································.104
5.4 esDNN：基于高效監(jiān)督學(xué)習(xí)的深度神經(jīng)網(wǎng)絡(luò) ··········································.106
5.4.1 多元時(shí)間序列預(yù)測(cè)的滑動(dòng)窗口 ····················································.106
5.4.2 esDNN算法 ···········································································.109
5.5 性能測(cè)試 ························································································.112
5.5.1 數(shù)據(jù)集和環(huán)境配置 ···································································.112
5.5.2 與基于無(wú)監(jiān)督學(xué)習(xí)方法的比較 ····················································.113
5.5.3 與基于神經(jīng)網(wǎng)絡(luò)方法的比較 ·······················································.114
5.5.4 與其他方面的比較 ···································································.118
本章小結(jié) ······························································································.119
思考題··································································································.120
參考文獻(xiàn) ······························································································.120
第6章 云應(yīng)用程序和可再生能源的自適應(yīng)管理方法 ····························.123
6.1 引言 ······························································································.124
6.2 相關(guān)工作 ························································································.125
6.2.1 DVFS和虛擬機(jī)整合·································································.125
6.2.2 Brownout ···············································································.126
6.2.3 數(shù)據(jù)中心冷卻系統(tǒng)的整體管理 ····················································.126
6.2.4 可再生能源 ············································································.126
6.3 系統(tǒng)模型 ························································································.127
6.4 問(wèn)題建模 ························································································.129
6.4.1 功率消耗 ···············································································.129
6.4.2 工作負(fù)載模型 ·········································································.130
6.4.3 優(yōu)化目標(biāo) ···············································································.131
6.5 根據(jù)可再生資源進(jìn)行調(diào)度決策 ····························································.132
6.5.1 Green-Aware 調(diào)度算法 ·····························································.132
6.5.2 交互式工作負(fù)載的Brownout算法 ···············································.133
6.5.3 批處理工作負(fù)載的延遲算法 ·······················································.134
6.5.4 主機(jī)調(diào)度 ···············································································.135
6.5.5 可再生能源預(yù)測(cè) ······································································.136
6.6 原型系統(tǒng)的實(shí)現(xiàn) ··············································································.137
6.7 性能評(píng)估 ························································································.139
6.7.1 環(huán)境設(shè)置 ···············································································.140
6.7.2 工作負(fù)載 ···············································································.140
6.7.3 應(yīng)用 ·····················································································.141
6.7.4 結(jié)果 ·····················································································.141
本章小結(jié) ·······························································································.145
思考題 ··································································································.145
參考文獻(xiàn) ·······························································································.146
第7章 云計(jì)算環(huán)境下基于虛擬機(jī)整合的自適應(yīng)節(jié)能調(diào)度 ······················.149
7.1 緒論 ······························································································.150
7.2 虛擬機(jī)整合技術(shù) ··············································································.150
7.3 相關(guān)研究工作 ··················································································.152
7.4 問(wèn)題定義 ························································································.153
7.5 數(shù)據(jù)中心能耗模型 ···········································································.154
7.5.1 服務(wù)器功耗模型 ······································································.154
7.5.2 服務(wù)器能耗模型 ······································································.155
7.5.3 云數(shù)據(jù)中心總能耗模型 ·····························································.156
7.5.4 數(shù)據(jù)中心節(jié)能調(diào)度下限 ·····························································.156
7.6 SAVE算法描述 ···············································································.158
7.6.1 概述 ·····················································································.158
7.6.2 分配算法 ···············································································.159
7.6.3 遷移算法 ···············································································.161
7.7 實(shí)驗(yàn)驗(yàn)證與分析 ··············································································.165
7.7.1 實(shí)驗(yàn)準(zhǔn)備 ···············································································.165
7.7.2 數(shù)據(jù)準(zhǔn)備 ···············································································.167
7.7.3 基線(xiàn)方法 ···············································································.167
7.7.4 結(jié)果分析 ···············································································.169
本章小結(jié) ······························································································.175
思考題··································································································.175
參考文獻(xiàn) ······························································································.175
第8章 MapReduce模型中數(shù)據(jù)傾斜問(wèn)題的算法 ·································.177
8.1 緒論 ······························································································.178
8.1.1 背景及意義 ············································································.178
8.1.2 研究現(xiàn)狀 ···············································································.179
8.1.3 研究?jī)?nèi)容 ···············································································.180
8.2 數(shù)據(jù)傾斜相關(guān)理論研究 ·····································································.181
8.2.1 數(shù)據(jù)傾斜 ···············································································.181
8.2.2 算法介紹 ···············································································.183
8.2.3 算法綜合對(duì)比 ·········································································.190
8.3 多任務(wù)數(shù)據(jù)傾斜調(diào)度算法設(shè)計(jì) ····························································.193
8.3.1 問(wèn)題描述與建模 ······································································.193
8.3.2 Revised Johnson1954算法（RJA） ··············································.195
8.3.3 離線(xiàn)多任務(wù)調(diào)度算法 ································································.199
8.3.4 在線(xiàn)多任務(wù)調(diào)度算法 ································································.201
8.4 單任務(wù)數(shù)據(jù)傾斜算法設(shè)計(jì) ··································································.203
8.4.1 YarnTune概述 ·········································································.203
8.4.2 檢測(cè)數(shù)據(jù)傾斜 ·········································································.206
8.4.3 YarnTune核心功能 ··································································.208
8.5 系統(tǒng)測(cè)試和分析 ··············································································.212
8.5.1 軟硬件測(cè)試環(huán)境 ······································································.212
8.5.2 多任務(wù)數(shù)據(jù)傾斜測(cè)試 ································································.212
8.5.3 單任務(wù)數(shù)據(jù)傾斜測(cè)試 ································································.217
本章小結(jié) ······························································································.220
思考題··································································································.221
參考文獻(xiàn) ······························································································.221
第9章 Spark中的數(shù)據(jù)均衡分配算法研究 ·········································.223
9.1 Spark設(shè)計(jì)思想 ················································································.224
9.1.1 Spark概述 ·············································································.224
9.1.2 Spark計(jì)算模型 ·······································································.225
9.1.3 Spark整體架構(gòu) ·······································································.226
9.2 Spark數(shù)據(jù)存儲(chǔ)體系 ··········································································.227
9.2.1 存儲(chǔ)整體架構(gòu) ·········································································.227
9.2.2 數(shù)據(jù)寫(xiě)入過(guò)程 ·········································································.228
9.2.3 數(shù)據(jù)讀取過(guò)程 ·········································································.229
9.3 Spark Shuffle分析 ············································································.230
9.3.1 Shuffle概述 ············································································.230
9.3.2 Shuffle寫(xiě)操作 ·········································································.231
9.3.3 Shuffle讀操作 ·········································································.232
9.4 Spark分區(qū)方法 ················································································.233
9.4.1 HashPartition分區(qū) ····································································.233
9.4.2 RangePartition分區(qū) ··································································.234
9.5 問(wèn)題描述與建模 ··············································································.235
9.5.1 相關(guān)定義 ···············································································.235
9.5.2 問(wèn)題建模 ···············································································.236
9.6 數(shù)據(jù)均衡分配算法整體設(shè)計(jì) ·······························································.237
9.6.1 抽樣算法 ···············································································.238
9.6.2 數(shù)據(jù)均衡分區(qū)算法 ···································································.240
9.6.3 權(quán)重調(diào)整算法 ·········································································.242
9.6.4 任務(wù)分配算法 ·········································································.245
9.7 算法復(fù)雜度分析 ··············································································.246
9.8 MRFair概述 ···················································································.246
9.8.1 MRFair的目標(biāo)與特征 ·······························································.246
9.8.2 MRFair系統(tǒng)架構(gòu) ·····································································.247
9.8.3 MRFair數(shù)據(jù)均衡分配示例 ·························································.248
9.9 MRFair傾斜檢測(cè)時(shí)機(jī)及算法 ······························································.249
9.9.1 MRFair傾斜檢測(cè)時(shí)機(jī) ·······························································.249
9.9.2 MRFair傾斜檢測(cè)算法 ·······························································.250
9.10 MRFair數(shù)據(jù)重新分配時(shí)機(jī)及算法 ······················································.250
9.10.1 MRFair數(shù)據(jù)重新分配時(shí)機(jī) ·······················································.250
9.10.2 MRFair數(shù)據(jù)重新分配算法 ·······················································.251
9.11 MRFair核心模塊 ············································································.253
9.12 系統(tǒng)測(cè)試環(huán)境 ················································································.255
9.12.1 軟硬件測(cè)試環(huán)境 ·····································································.255
9.12.2 測(cè)試數(shù)據(jù) ··············································································.256
9.12.3 對(duì)比算法或系統(tǒng) ·····································································.257
9.12.4 評(píng)價(jià)指標(biāo) ··············································································.257
9.13 Reduce Partition數(shù)據(jù)均衡分配算法測(cè)試 ··············································.257
9.13.1 WordCount基準(zhǔn)測(cè)試 ·······························································.257
9.13.2 Sort基準(zhǔn)測(cè)試 ········································································.260
9.14 MRFair數(shù)據(jù)均衡分配算法測(cè)試 ·························································.263
9.14.1 WordCount基準(zhǔn)測(cè)試 ·······························································.263
9.14.2 Sort基準(zhǔn)測(cè)試 ········································································.265
本章小結(jié) ······························································································.266
思考題··································································································.267
參考文獻(xiàn) ······························································································.267
第10章 深度學(xué)習(xí)框架TensorFlow的高效分布式并行算法研究 ·············.269
10.1 分布式并行算法的背景及意義 ···························································.270
10.1.1 問(wèn)題背景 ··············································································.270
10.1.2 研究意義 ··············································································.271
10.2 研究現(xiàn)狀及內(nèi)容 ·············································································.272
10.2.1 研究現(xiàn)狀 ··············································································.272
10.2.2 研究?jī)?nèi)容 ··············································································.272
10.3 深度學(xué)習(xí)理論研究 ··········································································.273
10.3.1 大數(shù)據(jù)與云計(jì)算 ····································································.273
10.3.2 機(jī)器學(xué)習(xí) ··············································································.274
10.3.3 深度學(xué)習(xí) ··············································································.275
10.4 TensorFlow深度學(xué)習(xí)框架研究 ··························································.277
10.4.1 TensorFlow系統(tǒng)架構(gòu) ······························································.277
10.4.2 TensorFlow數(shù)據(jù)流圖 ······························································.280
10.4.3 TensorFlow會(huì)話(huà)管理 ······························································.281
10.4.4 TensorFlow分布式執(zhí)行 ···························································.282
10.4.5 TensorFlow數(shù)據(jù)輸入 ······························································.283
10.5 TensorFlow分布式架構(gòu)分析 ·····························································.285
10.5.1 TensorFlow遠(yuǎn)程調(diào)用 ······························································.285
10.5.2 現(xiàn)有TensorFlow分布式模型 ····················································.286
10.6 優(yōu)化算法設(shè)計(jì)與實(shí)現(xiàn) ·······································································.289
10.6.1 數(shù)據(jù)并行上的優(yōu)化 ·································································.289
10.6.2 模型并行上的優(yōu)化 ·································································.297
10.7 實(shí)驗(yàn)環(huán)境配置 ················································································.304
10.7.1 硬件環(huán)境 ··············································································.304
10.7.2 軟件環(huán)境 ··············································································.304
10.7.3 實(shí)驗(yàn)對(duì)象 ··············································································.305
10.7.4 實(shí)驗(yàn)數(shù)據(jù) ··············································································.305
10.8 實(shí)驗(yàn)結(jié)果展示與分析 ·······································································.307
10.8.1 數(shù)據(jù)并行算法測(cè)試··································································.307
10.8.2 模型并行算法測(cè)試··································································.312
本章小結(jié) ·······························································································.314
思考題 ··································································································.314
參考文獻(xiàn) ·······························································································.314
第11章 基于深度強(qiáng)化學(xué)習(xí)和模仿學(xué)習(xí)的任務(wù)完工時(shí)間優(yōu)化調(diào)度 ···········.317
11.1 任務(wù)調(diào)度 ······················································································.318
11.2 相關(guān)研究 ······················································································.321
11.3 云資源調(diào)度問(wèn)題定義 ·······································································.322
11.4 調(diào)度方案 ······················································································.327
11.4.1 DeepRM_Online介紹 ······························································.327
11.4.2 強(qiáng)化學(xué)習(xí)模型 ········································································.328
11.4.3 深度強(qiáng)化學(xué)習(xí)訓(xùn)練算法····························································.330
11.5 算法分析 ······················································································.333
11.6 實(shí)驗(yàn)分析與驗(yàn)證 ·············································································.337
11.6.1 實(shí)驗(yàn)準(zhǔn)備 ··············································································.337
11.6.2 數(shù)據(jù)準(zhǔn)備 ··············································································.337
11.6.3 基線(xiàn)算法 ··············································································.338
11.6.4 結(jié)果分析 ··············································································.339
本章小結(jié) ·······························································································.341
思考題 ··································································································.341
參考文獻(xiàn) ·······························································································.342
第12章 總結(jié)與展望 ······································································.345
12.1 綠色節(jié)能數(shù)據(jù)中心的綜合解決方案·····················································.346
12.2 多數(shù)據(jù)中心（多調(diào)度域）的調(diào)度策略和算法動(dòng)態(tài)可選擇 ·························.348
12.3 支持深度學(xué)習(xí)模型的分布式并行調(diào)度 ·················································.349
12.4 從基礎(chǔ)資源調(diào)度拓展到應(yīng)用任務(wù)調(diào)度 ·················································.350
參考文獻(xiàn) ·······························································································.351