鏈枃灏嗘繁鍏ヨВ鏋愭繁搴﹀己鍖栧涔犱腑鐨勫叧閿粍浠垛€斺€擜ctor-Critic绠楁硶銆傝繖涓€绠楁硶缁撳悎浜嗙瓥鐣ヨ凯浠ｄ笌浠峰€艰凯浠ｇ殑浼樼偣锛屾槸寮哄寲瀛︿範棰嗗煙涓殑涓€椤归噸瑕侀泦鎴愮瓥鐣ャ€傞€氳繃瀹炶返绀轰緥锛屾垜浠皢灞曠ず濡備綍浣跨敤Pytorch瀹炵幇涓€涓畬鏁寸殑Actor-Critic妯″瀷銆?/p>

鎴戜滑鏉ヤ簡瑙ｄ竴涓婣ctor-Critic绠楁硶鐨勫熀鏈師鐞嗗拰缁勬垚閮ㄥ垎銆傝绠楁硶鍖呭惈涓や釜閮ㄥ垎锛氳鍔ㄨ€咃紙Actor锛夊拰璇勮瀹讹紙Critic锛夈€傝鍔ㄨ€呰礋璐ｉ€夋嫨鍔ㄤ綔锛屾棬鍦ㄦ渶澶у寲绱Н濂栧姳锛涜€岃瘎璁哄鍒欒瘎浼板綋鍓嶇瓥鐣ョ殑濂藉潖锛岀粰鍑哄姩浣滅殑浠峰€硷紝骞堕€氳繃鍙嶉鎸囧琛屽姩鑰呮ā鍨嬬殑鏇存柊銆傞€氳繃杩欑鏂瑰紡锛孉ctor-Critic绠楁硶涓嶆柇浼樺寲绛栫暐浠ユ渶澶у寲绱Н濂栧姳銆?/p>

鎺ヤ笅鏉ワ紝鎴戜滑灏嗚缁嗕粙缁岮ctor鍜岃瘎璁哄鐨勮鑹插拰鐩爣銆傝鍔ㄨ€呰緭鍑哄姩浣滅殑姒傜巼鍒嗗竷鎴栬繛缁姩浣滃€硷紝浠ユ渶澶у寲绱Н濂栧姳涓虹洰鏍囷紱璇勮瀹跺垯杈撳嚭鐘舵€佷环鍊兼垨鍔ㄤ綔浠峰€硷紝浠ヨ瘎浼板綋鍓嶇瓥鐣ョ殑濂藉潖骞朵紭鍖栬鍔ㄨ€呯殑琛屼负銆?/p>

鍦ㄥ叕寮忔帹瀵奸儴鍒嗭紝鎴戜滑灏嗕粙缁嶇瓥鐣ヤ紭鍖栫洰鏍囧嚱鏁般€佽瘎璁哄鎹熷け鍑芥暟鍜孉ctor鎹熷け鍑芥暟鐨勫叧閿叕寮忋€傜瓥鐣ヤ紭鍖栫殑鐩爣鏄渶澶у寲绱Н濂栧姳锛岃瘎璁哄缃戠粶閫氬父閲囩敤鐘舵€佷环鍊煎嚱鏁版垨鍔ㄤ綔浠峰€煎嚱鏁扮殑褰㈠紡锛屽叾鎹熷け鍑芥暟鍙互琛ㄧず涓篢D璇樊鐨勫钩鏂瑰潎鍊笺€傝€孉ctor缃戠粶鐨勭洰鏍囧垯鏄€氳繃鏈€澶у寲绛栫暐鐨勫鏁颁技鐒舵潵浼樺寲绛栫暐鍙傛暟銆傛垜浠繕灏嗕粙缁岮ctor鍜岃瘎璁哄鐨勫叧閿弬鏁版洿鏂板叕寮忋€?/p>

涓轰簡鏇寸洿瑙傚湴灞曠ずActor-Critic绠楁硶鐨勫簲鐢紝鎴戜滑灏嗗熀浜嶱ytorch搴撹繘琛屾搷浣溿€傛垜浠皢灞曠ず濡備綍鏋勯€犵瓥鐣ョ綉缁滐紙PolicyNet锛変笌浠峰€肩綉缁滐紙ValueNet锛夛紝骞跺疄鐜版ā鍨嬬殑鏇存柊鍜岃缁冭繃绋嬨€傛垜浠繕灏嗗埄鐢∣penAI Gym鐨凜artPole-v1鐜锛屾紨绀哄浣曞皢Actor-Critic绠楁硶搴旂敤浜庡疄闄呴棶棰樹腑銆?/p>

鍦ㄨ繖涓ず渚嬩腑锛屾垜浠皢灞曠ず瀛︿範鏇茬嚎浠ュ強姣忓洖鍚堢殑鍥炴姤锛屼互渚挎洿濂藉湴浜嗚В妯″瀷鐨勮缁冭繃绋嬪拰鎬ц兘銆傞€氳繃瀹為檯搴旂敤鍜岀粨鏋滃睍绀猴紝鎴戜滑灏嗗府鍔╄鑰呮洿濂藉湴鐞嗚ВActor-Critic绠楁硶鐨勫師鐞嗗拰瀹炵幇鏂瑰紡锛屽苟浜嗚В鍏跺湪瀹為檯闂涓殑琛ㄧ幇銆?/p>

---

3. 瀹氫箟绛栫暐缃戠粶锛圥olicyNet锛変笌鍐掗櫓涔嬫梾鍚▼

涓轰簡鎺㈢储鏈煡鐨勭幆澧冿紝鎴戜滑闇€瑕佷竴涓瓥鐣ョ綉缁滐紙PolicyNet锛夋潵鎸囧紩鎴戜滑鐨勮剼姝ャ€傝鎴戜滑瀹氫箟杩欎釜缃戠粶鐨勬瀯閫犲拰鍓嶈鏂瑰紡銆?/p>

绛栫暐缃戠粶锛圥olicyNet锛夌殑鏋勫缓涓庡惎绋嬩唬鐮侊細

```python

import torch

from torch import nn

class PolicyNet(nn.Module):

def __init__(self, 杈撳叆缁村害, 闅愯棌灞傜淮搴? 杈撳嚭缁村害):

super(PolicyNet, self).__init__() 缁ф壙nn.Module鐨勫垵濮嬪寲鏂规硶

self.fc1 = nn.Linear(杈撳叆缁村害, 闅愯棌灞傜淮搴? 绗竴灞傚叏杩炴帴缃戠粶锛岃礋璐ｇ壒寰佹彁鍙?/p>

self.fc2 = nn.Linear(闅愯棌灞傜淮搴? 杈撳嚭缁村害) 绗簩灞傚叏杩炴帴缃戠粶锛岃礋璐ｇ瓥鐣ヨ緭鍑烘鐜囧垎甯冮娴?/p>

self.鍚▼ = torch.relu 浣跨敤ReLU婵€娲诲嚱鏁板紑鍚垜浠殑鍐掗櫓涔嬫梾

def forward(self, 鐘舵€?: 瀹氫箟缃戠粶鐨勫墠鍚戜紶鎾繃绋嬶紝鍗崇瓥鐣ュ喅绛栬繃绋?/p>

鐘舵€佺粡杩囨縺娲诲嚱鏁板鐞嗗悗鐨勭壒寰?= self.鍚▼(self.fc1(鐘舵€?) 鐗瑰緛鎻愬彇闃舵锛屽緱鍒版縺娲诲悗鐨勭壒寰佸悜閲?/p>

鍐崇瓥姒傜巼鍒嗗竷 = torch.softmax(self.fc2(鐗瑰緛鍚戦噺), dim=1) 绛栫暐鍐崇瓥闃舵锛岃緭鍑烘瘡涓姩浣滅殑棰勬祴姒傜巼鍒嗗竷

return 鍐崇瓥姒傜巼鍒嗗竷 杩斿洖棰勬祴鐨勬鐜囧垎甯冿紝鎸囧紩鎴戜滑鐨勫喅绛栨柟鍚?/p>

```

鎺ヤ笅鏉ワ紝鎴戜滑灏嗗畾涔変环鍊肩綉缁滐紙ValueNet锛夛紝鐢ㄤ簬璇勪及褰撳墠鐘舵€佺殑浠峰€笺€傜劧鍚庯紝鎴戜滑灏嗚繖涓や釜缃戠粶缁勫悎鎴愪竴涓己澶х殑ActorCritic绫汇€傛渶缁堬紝鎴戜滑灏嗗湪涓€涓狢artPole鐜涓娇鐢ㄨ繖涓狝ctorCritic绫昏繘琛岃缁冨苟鍙鍖栫粨鏋溿€傝鎴戜滑缁х画鎺㈢储杩欎釜鍐掗櫓涔嬫梾鍚э紒馃殌馃幃馃敆鉁ㄥ湪杩欐浠ｇ爜鎵€鏋勫缓鐨勬儏澧冧腑锛屾垜浠繍鐢ㄤ簡涓€绉嶅熀鏈殑Actor-Critic妯″瀷锛屽畠琚簿蹇冪紪缁囧湪CartPole-v1鐜涓€傝繖涓幆澧冨鍚屼竴涓姩鎬佺殑鑸炲彴锛屽皬杞﹀拰鏉嗗瓙鍦ㄥ叾涓笉鏂簰鍔紝鑰屾垜浠殑妯″瀷鍒欏姫鍔涘浼氬浣曟帶鍒跺皬杞︾殑琛屽姩锛屼互淇濇寔鏉嗗瓙鐨勭珫鐩翠笉鍊掋€傝繖涓涔犺繃绋嬩笉浠呮槸涓€椤规妧鏈殑鎸戞垬锛屾洿鏄櫤鑳藉拰绛栫暐鐨勭鎾炪€傛瘡涓€娆＄殑琛屽姩閫夋嫨銆佹瘡涓€娆＄殑濂栧姳鍙嶉閮藉湪鎺ㄥ姩鐫€妯″瀷鍚戝墠杩涙銆傞€氳繃杩欎竴绯诲垪鐨勫皾璇曞拰淇锛屾ā鍨嬮€愭鎺屾彙浜嗕竴绉嶆妧鑳斤細鍦–artPole鐜涓紝濡備綍浠ユ渶浼樼殑鏂瑰紡琛屽姩浠ヨ幏鍙栨渶澶х殑绱Н濂栧姳銆傝€屾垜浠墍缁樺埗鐨勬洸绾匡紝姝ｆ槸妯″瀷瀛︿範杩囩▼涓殑鈥滄垚缁╁崟鈥濓紝瀹冩竻鏅板湴灞曠ず浜嗛殢鐫€璁粌鐨勮繘琛岋紝妯″瀷鎵€鑾峰緱鐨勭疮绉鍔辨槸濡備綍鍙樺寲鐨勩€傛瘡涓€娆＄殑杩唬锛岄兘鏄竴娆℃柊鐨勫皾璇曞拰绐佺牬锛屾瘡涓€娆＄殑濂栧姳鎻愬崌锛岄兘鏄ā鍨嬭繘姝ョ殑璇佹嵁銆傝繖涓繃绋嬪厖婊′簡鎸戞垬鍜屾儕鍠滐紝灏卞鍚屾垜浠皬鏃跺€欏涔犳柊鎶€鑳戒竴鏍凤紝铏界劧鏇叉姌鍗村厖婊′簡鎴愬氨鎰熴€傝繖灏辨槸浜哄伐鏅鸿兘瀛︿範鐨勯瓍鍔涙墍鍦紝涔熸槸鎴戜滑涓嶆柇杩芥眰杩涙鐨勬簮娉夈€?/p>