首頁(yè) 智能 > 正文

Docker 中的 .NET 異常了怎么抓 Dump|當(dāng)前獨(dú)家

2023-06-26 16:48:38 來(lái)源: 博客園
一：背景1. 講故事

有很多朋友跟我說(shuō)，在 Windows 上看過(guò)你文章知道了怎么抓 Crash, CPU爆高，內(nèi)存暴漲 等各種Dump，為什么你沒(méi)有寫(xiě)在 Docker 中如何抓的相關(guān)文章呢？瞧不上嗎？


(相關(guān)資料圖)

哈哈，在DUMP的分析旅程中，跑在 Docker 中的 .NET 占比真的不多，大概10個(gè)dump有 1-2 個(gè)是 docker 中的，市場(chǎng)決定了我的研究方向，為了彌補(bǔ)這一塊的空洞，決定寫(xiě)一篇文章來(lái)分享下這三大異常下的捕獲吧。

二：Docker 下的三大異常捕獲1. crash dump 捕獲

前不久我寫(xiě)了一篇 Linux 上的 .NET 崩潰了怎么抓 Dump （https://www.cnblogs.com/huangxincheng/p/17440153.html）的文章，使用了微軟推薦的環(huán)境變量方式，其實(shí)這在 Docker 中是一樣適用的。

為了讓 webapi崩潰退出，我故意造一個(gè)棧溢出異常，參考代碼如下：

public class Program    {        public static void Main(string[] args)        {            var builder = WebApplication.CreateBuilder(args);            builder.Services.AddAuthorization();            var app = builder.Build();            app.UseAuthorization();            //1. crash            Task.Factory.StartNew(() =>            {                Test("a");            });            app.Run();        }        public static string Test(string a)        {            return Test("a" + a.Length);        }    }

有了代碼之后，接下來(lái)寫(xiě)一個(gè) Dockerfile，主要就是把三個(gè)環(huán)境變量塞進(jìn)去。

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS runtimeWORKDIR /appCOPY ./ ./# 1. 使用中科大鏡像源RUN sed -i "s/deb.debian.org/mirrors.ustc.edu.cn/g" /etc/apt/sources.listENV COMPlus_DbgMiniDumpType 4ENV COMPlus_DbgMiniDumpName /dumps/%p-%e-%h-%t.dmpENV COMPlus_DbgEnableMiniDump 1ENTRYPOINT ["dotnet", "AspNetWebApi.dll"]

這里有一個(gè)細(xì)節(jié)，為了能夠讓 Docker 中的 webapi 能夠訪問(wèn)到，將 localhost 設(shè)置為 * ，修改 appsettings.json如下：

{  "urls": "http://*:5001",  "Logging": {    "LogLevel": {      "Default": "Information",      "Microsoft.AspNetCore": "Warning"    }  },  "AllowedHosts": "*"}

有了這些基礎(chǔ)最后就是 docker build & docker run 啦。

[root@localhost data]# docker build -t aspnetapp .[+] Building 0.3s (9/9) FINISHED                                                                          => [internal] load build definition from Dockerfile                                                0.0s => => transferring dockerfile: 447B                                                                0.0s => [internal] load .dockerignore                                                                   0.0s => => transferring context: 2B                                                                     0.0s => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:6.0                                0.3s => [1/4] FROM mcr.microsoft.com/dotnet/aspnet:6.0@sha256:a2a04325fdb2a871e964c89318921f82f6435b54  0.0s => [internal] load build context                                                                   0.0s => => transferring context: 860B                                                                   0.0s => CACHED [2/4] WORKDIR /app                                                                       0.0s => CACHED [3/4] COPY ./ ./                                                                         0.0s => CACHED [4/4] RUN sed -i "s/deb.debian.org/mirrors.ustc.edu.cn/g" /etc/apt/sources.list          0.0s => exporting to image                                                                              0.0s => => exporting layers                                                                             0.0s => => writing image sha256:be69203995c0e5423b2af913549e618d7ee8306fff3961118ff403b1359ae571        0.0s => => naming to docker.io/library/aspnetapp                                                        0.0s[root@localhost data]# docker run -itd  -p 5001:5001 --privileged -v /data2:/dumps --name aspnetcore_sample aspnetappca34c9274d998096f8562cbef3a43a7cbd9aa5ff2923e0f3e702b159e0b2f447[root@localhost data]# docker ps -aCONTAINER ID   IMAGE       COMMAND                  CREATED          STATUS                       PORTS     NAMESca34c9274d99   aspnetapp   "dotnet AspNetWebApi…"   20 seconds ago   Exited (139) 9 seconds ago             aspnetcore_sample[root@localhost data]# docker logs ca34c9274d99   ...   at AspNetWebApi.Program.Test(System.String)   at AspNetWebApi.Program.Test(System.String)   at AspNetWebApi.Program.Test(System.String)   at AspNetWebApi.Program.Test(System.String)   at AspNetWebApi.Program+<>c.
b__0_0()   at System.Threading.Tasks.Task.InnerInvoke()   at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object)   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)   at System.Threading.ThreadPoolWorkQueue.Dispatch()   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()   at System.Threading.Thread.StartCallback()[createdump] Gathering state for process 1 dotnet[createdump] Crashing thread 0017 signal 6 (0006)[createdump] Writing full dump to file /dumps/1-dotnet-ca34c9274d99-1687746929.dmp[createdump] Written 261320704 bytes (63799 pages) to core file[createdump] Target process is alive[createdump] Dump successfully written[root@localhost data2]# cd /data2[root@localhost data2]# ls -lntotal 255288-rw-------. 1 0 0 261414912 Jun 26 10:35 1-dotnet-ca34c9274d99-1687746929.dmp

上面的腳本已經(jīng)寫(xiě)的非常清楚了，這里有幾個(gè)注意點(diǎn)提一下：

--privileged

一定要加上特殊權(quán)限，否則生成 dump 的時(shí)候會(huì)提示無(wú)權(quán)限。

-v /data2:/dumps

防止dump丟失，記得掛載到宿主機(jī)目錄 或者 共享容器 中。

2. 內(nèi)存暴漲 dump 捕獲

要想對(duì) docker 中的 .NET 程序內(nèi)存 進(jìn)行監(jiān)控，我一直都是極力推薦 procdump，目前最新的是版本是 1.5， github官網(wǎng)地址： https://github.com/Sysinternals/ProcDump-for-Linux 鑒于現(xiàn)在訪問(wèn) github 太慢，大家可以把 procdump_1.5-16239_amd64.deb下載到本地，為什么下載它，是因?yàn)槿萜髦惺?debain 系統(tǒng)。

下載好了之后放到項(xiàng)目中，使用默認(rèn)代碼骨架：

public class Program    {        public static void Main(string[] args)        {            var builder = WebApplication.CreateBuilder(args);            builder.Services.AddAuthorization();            var app = builder.Build();            app.UseAuthorization();            app.Run();        }    }

接下來(lái)就是寫(xiě) dockerfile 了，這里有一個(gè)細(xì)節(jié)，就是如何在 Docker 中開(kāi)啟多進(jìn)程，這里用 start.sh 腳本的方式開(kāi)啟，參考代碼如下：

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS runtimeWORKDIR /appCOPY ./ ./# 1. 使用中科大鏡像源RUN sed -i "s/deb.debian.org/mirrors.ustc.edu.cn/g" /etc/apt/sources.list# 2. 安裝 gdb & procdumpRUN apt-get update && apt-get install -y gdbRUN dpkg -i procdump.debRUN echo "#!/bin/bash \n\procdump -m 30 -w dotnet /dumps & \n\dotnet \$1 \n\" > ./start.shRUN chmod +x ./start.shENTRYPOINT ["./start.sh", "AspNetWebApi.dll"]

有了這些設(shè)置后，接下來(lái)就是 publish 代碼用 docker 構(gòu)建啦，為了方便演示，這里就用 前臺(tái)模式開(kāi)啟了哈。

[root@localhost data]# docker build -t aspnetapp .[+] Building 11.5s (13/13) FINISHED              [root@localhost data]# docker rm -f aspnetcore_sampleaspnetcore_sample[root@localhost data]# docker run -it --rm  -p 5001:5001 --privileged -v /data2:/dumps --name aspnetcore_sample aspnetappProcDump v1.5 - Sysinternals process dump utilityCopyright (C) 2023 Microsoft Corporation. All rights reserved. Licensed under the MIT license.Mark Russinovich, Mario Hewardt, John Salem, Javid HabibiSysinternals - www.sysinternals.comMonitors one or more processes and writes a core dump file when the processes exceeds thespecified criteria.[02:57:34 - INFO]: Waiting for processes "dotnet" to launch[02:57:34 - INFO]: Press Ctrl-C to end monitoring without terminating the process(es).Process Name:                           dotnetCPU Threshold:                          n/aCommit Threshold:                       >=30 MBThread Threshold:                       n/aFile Descriptor Threshold:              n/aSignal:                                 n/aException monitor                       OffPolling Interval (ms):                  1000Threshold (s):                          10Number of Dumps:                        1Output directory:                       /dumps[02:57:34 - INFO]: Starting monitor for process dotnet (9)info: Microsoft.Hosting.Lifetime[14]      Now listening on: http://[::]:5001info: Microsoft.Hosting.Lifetime[0]      Application started. Press Ctrl+C to shut down.info: Microsoft.Hosting.Lifetime[0]      Hosting environment: Productioninfo: Microsoft.Hosting.Lifetime[0]      Content root path: /app/[02:57:35 - INFO]: Trigger: Commit usage:48MB on process ID: 9[createdump] Gathering state for process 9 dotnet[createdump] Writing full dump to file /dumps/dotnet_commit_2023-06-26_02:57:35.9[createdump] Written 254459904 bytes (62124 pages) to core file[createdump] Target process is alive[createdump] Dump successfully written[02:57:35 - INFO]: Core dump 0 generated: /dumps/dotnet_commit_2023-06-26_02:57:35.9[02:57:36 - INFO]: Stopping monitors for process: dotnet (9)[root@localhost data2]# ls -lhtotal 243M-rw-------. 1 root root 243M Jun 26 10:57 dotnet_commit_2023-06-26_02:57:35.9

從腳本信息看，當(dāng)內(nèi)存到了 48MB的時(shí)候觸發(fā)的 dump 生成，也成功的進(jìn)入了 /dumps目錄中，太棒了。

3. cpu爆高 dump 捕獲

抓 cpu 爆高的dump最好的方式就是多抓幾個(gè)，比如說(shuō)：當(dāng) CPU >20% 連續(xù)超過(guò) 5s 抓 2個(gè)dump，這種方式抓的dump很容易就能找到真兇，為了方便演示，讓兩個(gè) cpu 直接打滿，參考代碼如下：

public static void Main(string[] args)        {            var builder = WebApplication.CreateBuilder(args);            builder.Services.AddAuthorization();            var app = builder.Build();            app.UseAuthorization();            //3. cpu            app.MapGet("/cpu", (HttpContext httpContext) =>            {                Task.Factory.StartNew(() => { bool b = true; while (true) { b = !b; } });                Task.Factory.StartNew(() => { bool b = true; while (true) { b = !b; } });                return new WeatherForecast();            });            app.Run();        }

接下來(lái)就是修改 dockerfile，因?yàn)槲业奶摂M機(jī)是 8 核心，如果兩個(gè)核心被打滿，那應(yīng)該會(huì)占用大概 24% 的 cpu 利用率，所以腳本中就設(shè)置 20% 吧。

FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS runtimeWORKDIR /appCOPY ./ ./# 1. 使用中科大鏡像源RUN sed -i "s/deb.debian.org/mirrors.ustc.edu.cn/g" /etc/apt/sources.list# 2. 安裝 wgetRUN apt-get update && apt-get install -y gdbRUN dpkg -i procdump.debRUN echo "#!/bin/bash \n\procdump -c 20 -n 2 -s 5 -w dotnet /dumps & \n\dotnet \$1 \n\" > ./start.shRUN chmod +x ./start.shENTRYPOINT ["./start.sh", "AspNetWebApi.dll"]

最后就是 docker 構(gòu)建。

[root@localhost data]# docker build -t aspnetapp .[+] Building 0.4s (13/13) FINISHED[root@localhost data]# docker run -it --rm  -p 5001:5001 --privileged -v /data2:/dumps --name aspnetcore_sample aspnetappProcDump v1.5 - Sysinternals process dump utilityCopyright (C) 2023 Microsoft Corporation. All rights reserved. Licensed under the MIT license.Mark Russinovich, Mario Hewardt, John Salem, Javid HabibiSysinternals - www.sysinternals.comMonitors one or more processes and writes a core dump file when the processes exceeds thespecified criteria.[03:35:56 - INFO]: Waiting for processes "dotnet" to launch[03:35:56 - INFO]: Press Ctrl-C to end monitoring without terminating the process(es).Process Name:                           dotnetCPU Threshold:                          >= 20%Commit Threshold:                       n/aThread Threshold:                       n/aFile Descriptor Threshold:              n/aSignal:                                 n/aException monitor                       OffPolling Interval (ms):                  1000Threshold (s):                          5Number of Dumps:                        2Output directory:                       /dumps[03:35:56 - INFO]: Starting monitor for process dotnet (8)info: Microsoft.Hosting.Lifetime[14]      Now listening on: http://[::]:5001info: Microsoft.Hosting.Lifetime[0]      Application started. Press Ctrl+C to shut down.info: Microsoft.Hosting.Lifetime[0]      Hosting environment: Productioninfo: Microsoft.Hosting.Lifetime[0]      Content root path: /app/

看輸出是正在監(jiān)控，接下來(lái)我們?cè)L問(wèn)下網(wǎng)址： http://192.168.17.129:5001/cpu，稍等片刻之后就會(huì)生成兩個(gè)dump 文件。

三：總結(jié)

雖然Docker中的 .NET 程序占比較少，但把經(jīng)驗(yàn)總結(jié)出來(lái)還是很值得的，以后有人問(wèn)怎么抓，可以把這篇文章直接丟過(guò)去啦！

關(guān)鍵詞：

最近更新

關(guān)于本站 管理團(tuán)隊(duì) 版權(quán)申明 網(wǎng)站地圖 聯(lián)系合作 招聘信息

Copyright © 2005-2023 創(chuàng)投網(wǎng) - m.670818.com All rights reserved
聯(lián)系我們:39 60 29 14 2@qq.com
皖I(lǐng)CP備2022009963號(hào)-3