銆銆濡備粖澶ф暟鎹鍜屾満鍣ㄥ︿範宸茬粡鏈変簡寰堝ぇ鐨勭粨鍚堬紝鍦ㄦ満鍣ㄥ︿範閲岄潰锛屽洜涓鸿＄畻杩浠ｇ殑鏃堕棿鍙鑳戒細寰堥暱锛屽紑鍙戜汉鍛樹竴鑸浼氶夋嫨浣跨敤 GPU銆丗PGA 鎴 TPU 鏉ュ姞閫熻＄畻銆傚湪 Apache Hadoop 3.1 鐗堟湰閲岄潰宸茬粡寮濮嬪唴缃鍘熺敓鏀鎸 GPU 鍜 FPGA 浜嗐備綔涓洪氱敤璁＄畻寮曟搸鐨 Spark 鑲瀹氫篃涓嶇敇钀藉悗锛屾潵鑷 Databricks銆丯VIDIA銆丟oogle 浠ュ強闃块噷宸村反鐨勫伐绋嬪笀浠姝ｅ湪涓 Apache Spark 娣诲姞鍘熺敓鐨 GPU 璋冨害鏀鎸侊紝璇ユ柟妗堝～琛ヤ簡 Spark 鍦 GPU 璧勬簮鐨勪换鍔¤皟搴︽柟闈㈢殑绌虹櫧锛屾湁鏈哄湴铻嶅悎浜嗗ぇ鏁版嵁澶勭悊鍜 AI 搴旂敤锛屾墿灞曚簡 Spark 鍦ㄦ繁搴﹀︿範銆佷俊鍙峰勭悊鍜屽悇澶ф暟鎹搴旂敤鐨勫簲鐢ㄥ満鏅銆傝繖椤瑰伐浣滅殑 issue 鍙浠ュ湪 SPARK-24615 閲岄潰鏌ョ湅锛岀浉鍏崇殑 SPIP(Spark Project Improvement Proposals) 鏂囨。鍙浠ュ弬瑙 SPIP: Accelerator-aware scheduling

銆銆

銆銆濡傛灉鎯冲強鏃朵簡瑙Spark銆丠adoop鎴栬匟base鐩稿叧鐨勬枃绔狅紝娆㈣繋鍏虫敞寰淇″叕鍏卞笎鍙凤細iteblog_hadoop

銆銆鐩鍓 Apache Spark 鏀鎸佺殑璧勬簮绠＄悊鍣 YARN 鍜 Kubernetes 宸茬粡鏀鎸佷簡 GPU銆備负浜嗚 Spark 涔熸敮鎸 GPUs锛屽湪鎶鏈灞傞潰涓婇渶瑕佸仛鍑轰袱涓涓昏佹敼鍙

銆銆鍦 cluster manager 灞傞潰涓婏紝闇瑕佸崌绾 cluster managers 鏉ユ敮鎸 GPU銆傚苟涓旂粰鐢ㄦ埛鎻愪緵鐩稿叧 API锛屼娇寰楃敤鎴峰彲浠ユ帶鍒 GPU 璧勬簮鐨勪娇鐢ㄥ拰鍒嗛厤銆

銆銆鍦 Spark 鍐呴儴锛岄渶瑕佸湪 scheduler 灞傞潰鍋氬嚭淇鏀癸紝浣垮緱 scheduler 鍙浠ュ湪鐢ㄦ埛 task 璇锋眰涓璇嗗埆 GPU 鐨勯渶姹傦紝鐒跺悗鏍规嵁 executor 涓婄殑 GPU 渚涚粰鏉ュ畬鎴愬垎閰嶃

銆銆鍥犱负璁 Apache Spark 鏀鎸 GPU 鏄涓涓姣旇緝澶х殑鐗规э紝鎵浠ラ」鐩鍒嗕负浜嗗嚑涓闃舵点傚湪 Apache Spark 3.0 鐗堟湰锛屽皢鏀鎸佸湪 standalone銆 YARN 浠ュ強 Kubernetes 璧勬簮绠＄悊鍣ㄤ笅鏀鎸 GPU锛屽苟涓斿圭幇鏈夋ｅ父鐨勪綔涓氬熀鏈娌″奖鍝嶃傚逛簬 TPU 鐨勬敮鎸併丮esos 璧勬簮绠＄悊鍣ㄤ腑 GPU 鐨勬敮鎸併佷互鍙 Windows 骞冲彴鐨 GPU 鏀鎸佸皢涓嶆槸杩欎釜鐗堟湰鐨勭洰鏍囥傝屼笖瀵逛簬涓寮 GPU 鍗″唴鐨勭粏绮掑害璋冨害涔熶笉浼氬湪杩欎釜鐗堟湰鏀鎸;Apache Spark 3.0 鐗堟湰灏嗘妸涓寮 GPU 鍗″拰鍏跺唴瀛樹綔涓轰笉鍙鍒嗗壊鐨勫崟鍏冦

銆銆瀹炵幇姒傛嫭

銆銆Spark Scheduling

銆銆鍦ㄨ繖涓灞傞潰锛屾垜浠寰楀厑璁镐粠 RDD/PandasUDF API 涓鎸囧畾璧勬簮璇锋眰锛岃繖浜涜锋眰搴旇ュ湪 DAGScheduler 涓姹囨汇俆askSetManager 绠＄悊姣忎釜 Stage 鎸傝捣(pending)鐨勪换鍔★紝瀵逛簬閭ｄ簺鏈 GPU 璇锋眰鐨勪换鍔★紝鎴戜滑闇瑕佸勭悊;瀵逛簬閭ｄ簺涓嶉渶瑕 GPU 鐨勪綔涓氾紝鍏惰皟搴﹁屼负鍜屾晥鐜囧簲璇ュ拰涔嬪墠淇濇寔涓鑷淬

銆銆鐩鍓嶏紝CPUS_PER_TASK(spark.task.cpus)鏄涓涓 int 绫诲瀷鐨勫叏灞閰嶇疆锛岀敤浜庢寚瀹氭瘡涓 task 搴斿垎閰嶇殑 cores銆備负浜嗘敮鎸 GPU 鐨勯厤缃锛屽紩鍏ヤ簡 spark.task.gpus 鍙傛暟鐢ㄤ簬鎸囧畾姣忎釜 task 闇瑕佺敵璇风殑 GPU 鏁般傚傛灉鐢ㄦ埛娌℃湁鎸囧畾 spark.task.cpus 鎴 spark.task.gpus锛岄偅涔 Spark 绋嬪簭灏嗕娇鐢ㄩ粯璁ょ殑鍊;鍥犱负闇瑕佸悜鍚庡吋瀹癸紝鎵浠ュ傛灉鐢ㄦ埛娌℃寚瀹 spark.task.cpus 鎴 spark.task.gpus锛岃繖涓や釜鍙傛暟鐨勯粯璁ゅ煎垎鍒涓 1 鍜 绌恒

銆銆瀵逛簬 ExecutorBackend 锛岄渶瑕佷娇寰楀畠鍙浠ヨ瘑鍒鍜岀＄悊 GPU 锛屽苟涓旀妸杩欎簺淇℃伅鍚屾(姣斿備慨鏀圭幇鏈夌殑 RegisterExecutor 绫)鍒 SchedulerBackend锛岀劧鍚 SchedulerBackend 鍙浠ユ牴鎹杩欎簺 GPU 淇℃伅锛屼负閭ｄ簺闇瑕 GPU 璧勬簮鐨 task 杩涜岃祫婧愬垎閰嶃

銆銆Resource Manager

銆銆绗涓闃舵靛皢鍦 Standalone銆乊ARN 浠ュ強 Kubernetes 涓婃敮鎸 GPU銆係park 闇瑕佸湪杩欎笁绉嶈祫婧愮＄悊涓婇潰鍋氫竴浜涘伐浣溿

銆銆Standalone

銆銆Standalone 鏄 Spark 鍐呯疆鐨勮祫婧愮＄悊妯″紡锛屼絾鏄鐩鍓嶇殑 Standalone 閮ㄧ讲妯″紡骞朵笉鑳芥敮鎸 GPU 绛夎祫婧愩備负浜嗚兘璇嗗埆 GPU 淇℃伅锛屼竴绉嶅彲琛岀殑鏂规硶鏄鍦ㄩ厤缃鏂囦欢閲岄潰瀵 GPU 璧勬簮杩涜岄厤缃锛 Worker 閫氳繃璇诲彇杩欎簺閰嶇疆淇℃伅锛屽苟鍦ㄥ唴瀛樼粨鏋勯噷闈㈢淮鎶 GPU 鍜 CPU 绛夊彲鐢ㄨ祫婧愮瓑淇℃伅銆傚悓鏃讹紝鍦 Master 涓婇氳繃 allocateWorkerResourceToExecutors 鏂规硶瀵 Executors 鐢宠风殑璧勬簮(鍖呮嫭 GPU)杩涜屽垎閰嶃

銆銆YARN

銆銆涓轰簡鑳藉熷湪 YARN 涓婃敮鎸 GPU锛屾垜浠闇瑕佷娇鐢 YARN 3.1.2+ 鐗堟湰;鍚屾椂鎴戜滑闇瑕佸湪 YARN 闆嗙兢涓婂仛鍑虹浉鍏抽厤缃锛屼娇寰 YARN 鍚鍔ㄤ簡瀵 GPU 璧勬簮鐨勬敮鎸侊紝鍏充簬濡備綍鍦 YARN 涓婇厤缃 GPU 璧勬簮锛岃峰弬瑙佽繖閲屻

銆銆褰撲负 Executors 鐢宠 YARN 瀹瑰櫒鏃讹紝Spark 闇瑕佸湪 YARN 瀹瑰櫒璇锋眰涓灏 executor 鎵闇鐨 GPU 鏁伴噺鏄犲皠鍒 yarn.io/gpu 璧勬簮涓銆俌ARN 鍏锋湁 GPU 闅旂绘満鍒讹紝鎵浠ユ棤璁烘槸鍚︿娇鐢 Docker 瀹瑰櫒锛 瀵规湭鍒嗛厤缁 YARN 瀹瑰櫒鐨 GPU 璧勬簮鐨勪娇鐢ㄥ皢浼氳闃绘銆

銆銆闇瑕佹敞鎰忕殑鏄锛屾埅鑷崇洰鍓 YARN 浠呮敮鎸 Nvidia GPU銆

銆銆Kubernetes

銆銆浠 Kubernetes 1.8 鐗堟湰寮濮嬶紝Kubernetes 浣跨敤璁惧囨彃浠舵ā鍨(device plugin model)鏉ユ敮鎸 GPU銆侀珮鎬ц兘NIC锛孎PGA 绛夎惧囥傜洰鍓 Kubernetes 鏀鎸 Nvidia 銆丄MD 鍜 Intel 鐨 GPU 璁惧囥傚湪 Spark + k8s 閲岄潰涓 task 鎸囧畾 GPU 鐨勬暟閲忓拰鍦 Standalone 鎴 YARN 妯″紡閲岄潰涓鏍枫備篃鏄鏀鎸 spark.task.gpus 鍜 spark.executor.gpus 鐨勫叏灞閰嶇疆锛屼篃鏀鎸佸湪 RDD stage 涓涓烘瘡涓 task 璁剧疆銆