Greg Suvalian's blog

Using URLRewrite to curcumvent limitation of anonymous WebDAV in IIS7

2012-02-09T10:52:00.011-06:00

IIS7 removed possibility to use anonymous WebDAV and one of the solution proposed to curcumvent this (Here) was using custom HTTP Module which would inject Basic Authorization header into the stream.

This works but pretty difficult to implement. First you have to compile assembly, secondly you have to hardcode username and password into DLL without possibility of changing it once deployed. There are some drawbacks to that solution as well since that module is being executed on every incoming request regardless wether it's WebDAV or not.
Instead of that you can just simple use URLRewrite module to do exactly the thing mentioned with no additional complexities mentioned. You need to create rewrite configuration below. You encode desired username password using base64 encoding (for example here Here) in format "username:password" and append that value after "Basic" value below.

Managed Stack Explorer compatible with x64 and IIS 7 environment

2010-04-27T13:40:00.003-05:00

Hi,

This is just recompiled version of Managed Stack Explorer which will run in x64 environment and IIS7.
File is here for download

Virtual CPUs and performance of virtual machine

2010-03-05T12:46:00.001-06:00

Performance comparison of multi threaded application in relevance to multiple virtual CPUs in virtual machine

I was tasked (actually wanted to know myself) if common understanding going around that adding virtual CPUs to virtual machine does not affect performance of such. Idea behind this thought is that since CPU is virtual then hypervisor will be spreading load across physical CPUs anyway and hence additional CPUs will not increase performance of your application.
I wanted specifically to test multi threaded application since this is what majority of web servers are.
So to test raw CPU performance I acquired 4 CPU IBM box. I wanted to make a test against VM guest running inside ESX 3.5 with different number of virtual CPUs and 1 another test - against physical hardware (no hypervisor).
For test itself I wanted something which is very CPU intensive and has no IO dependencies. I figured calculating PI to certain digit will be a good choice in this case.
Entire Visual Studio 2008 solution can be downloaded below. It consists of web service which accepts only 1 parameter which is "Number of Digits of PI to be calculated to" and returns back total time taken to calculate that number.
Solution also contains sample webpage which will call web service, that page was called by 10 virtual clients simultaneously. Total number of pages served were calculated and presented in table and graph below.

Conclusion: Adding more virtual CPUs for multithreaded application does it in fact increase performance of application.

Number of logical CPUs	Number of requests served
1 CPU	24
2 CPU	47
4 CPU	65
Physical box with 4 CPUs	70

Attachment:Download

How to use .NET profiler to troubleshoot ASP.NET page perfomance issues

2009-05-22T14:18:00.002-05:00

How to use .NET profiler to troubleshoot slow ASP.NET page performance

I'm frequently tasked to figure out why certain pages or parts of websites are slow (assuming site is ASP.NET based). In this specific article I'll show how to use free .NET profiler from Equatec http://www.eqatec.com/tools/profiler to troubleshoot this issues.

My common troubleshooting steps (before diving into ASP.NET website performance troubleshooting are below).

Use Fiddler tool (http://www.fiddler2.com/fiddler2/) to actually measure response time of web page. This works with AJAX based pages as well and will actually provide you information which you usually would not be able to see in browser window (simple GET/POST requests are so 90s in modern technologies).
Once you get statistics and confirmed that you do in fact have backend problem other then networking issues I usually see if there are any known dependencies for the page (like SQL backend). This is pretty easy to figure out by looking through web.config file. I start SQL profiler on SQL server afterwards and resubmit request from browser again watching profiler statements coming through. Sometimes troubleshooting ends here (if you see that Execution time of stored procedure or ad-hoc query corresponds to total page response time.
At this point if you ruled out backend slowness you are probably looking at some issues within ASP.NET website itself. I usually enable ASP.NET tracing at this point by going to web.config and adding following statement there

<trace enabled="true" localOnly="false" pageOutput="true"/>). This will output tracing information (most importanly times to takes to execute specific methods in pipeline). Sometimes it's enough to to see which part of the page pipeline fails.
Please note that page execution times listed trace output are not including times which are spends outside of the page (like authenticating request or processing events in global.asax). So you might see fast execution times in trace output but page will still be slow to render and hence we come to real purpose of this article, i.e. to use .NET profiler to see what is causing page to stall.

I wrote a sample website where I simulated different stalling conditions. Stalling in pre-page execution pipeline (in global.asax) and stalling inside page itself (SQL call or Web Services call or something of that nature). To simulate long running method I inserted following statement in relevant methods (System.Threading.Thread.Sleep(5000);). This makes current thread sleep for specified amount of time. I put Tracing statements through the code so you can actually see in screenshot below where latency are being introduced. Lines in red indicate Trace statements in the code. (First one in global.asax Application_BeginRequest event and second one inside Page_Load() statement of default.aspx page). In real life scenario you most likely would not have any Tracing statements in code you'll be troubleshooting though.

I compiled my test website and at this point all I have is what you probably would have in your case. You can download it here (ZIP). Once you installed it, navigate to default.aspx page and see how much it takes to load a page (shall be around 7s or so).

Next step is to install Equatec .NET profiler from here (Here).

On first run it'll display default screen below

Click browse button and navigate to installation directory of your website (BIN folder) and open precompiled binary for your application (profiler.dll in this case)

Click "Build" button on bottom. Profiler will inject profiling information into your assembly and output results to the "Bin-Profiled" folder under the same root where your original "Bin" folder is. You'll see that you original DLL file grew up in size because of that.

Copy all files from "Bin-Profiled" folder to "Bin" folder (replacing files)

Switch to "Run" tab and navigate to your problem page. You'll notice that Control area will be updated with information "Profiled app Profiler started"

Click on "Take snapshot" button. It'll generate report about execution time and put report under "View Snapshot reports" area.

Click "View" button now with latest report selected. Result below is exactly what you'd expect to see from test file. Which is first 5 seconds were spent in BeginRequest event and next 2 in Page_Load calling "DoSQLQuery" method.

Now since you know which specifics methods take that much time you need to take a look at what is being done inside those methods. In this case (and probably yours) you'll just have assembly (profile.dll). You need to use tool like .NET Reflector (Download here) to see contents in readable form.

Now we can see exactly why it took so long to execute this specific method. In your case it might be SQL call or web services or call or something of that nature.

Instructions how to enable SSTP VPN s...

2009-04-06T11:43:00.001-05:00

Instructions how to enable SSTP VPN server in Windows 2008 simplified

In this brief 10 minutes procedure below you'll learn how to create VPN connection using SSTP (SSL tunnel) to your work/home location from anywhere where HTTPS is allowed.

Guide below assumes you don't want to use your VPN server as Internet Router and you have some other devices doing routing. Steps below will allow you to VPN in your remote location and route everything (including Internet traffic) through that VPN connection.

Difference from other guides on Internet:

Don't need domain controller or certificate services for this to work
Don't need 2 network adapters on your server to work
Don't need to publish SSL CRL (certificate revocation list)

What is needed:

Server SSL certificate (self-signed)
Windows 2008 on your local network (single NIC)
Vista SP1 or later

Steps:

Create self-signed certificate and import it into certificate store on client and server
- Create self-signed certificate with domain name which you'll be using for your VPN server. I used this excellent utility (http://www.pluralsight.com/community/blogs/keith/archive/2009/01/22/create-self-signed-x-509-certificates-in-a-flash-with-self-cert.aspx). What you want to do is to put CN name as external name which you'll be using for your VPN server (like sstp.mydomain.com) and choose to export as PFX file. This will create PFX file with your certificate and designated location. In steps below it would be imported into client and server as Trusted CA.
  - Import certificate on client computer by following this steps: Launch MMC and choose Add Snap-in, choose "Certificates", on next screen choose "Local Computer" and then choose "Trusted Root Certification Authorities" and import your certificate into that store.
  - Import certificate on server computer exactly the same ways it was done on client.
Enabled RRAS role on Windows 2008 and configure SSTP.
- Install RRAS service on your server
  - Add Server Role called "Network Policy and Access Services". Choose only "Remote Access Service" for service since we don't need routing. Go through installation process.
  - Once server is setup, launch Routing and Remote Access MMC and choose to "Configure Routing and Remote Access server".
  - Choose "Custom Configuration" and choose "VPN access".
  - For your user go to user property pages. Choose "Dial-in" tab and choose "Allow access" under "Network Access Permission".
- Configure your external router to port forward TCP 443 to your internal IP of your VPN server.
Configure your Vista VPN client.
- Go to Network And Sharing Center/Setup Connection or network/Connect to a workplace. Choose "Use Internet Connection", put your hostname for connection and go through the rest of the steps. Use "skip" since connection will probably fail since we need to configure connection to use SSTP. Once connection is created, go to properties of connection/Networking and choose as type of VPN - SSTP, uncheck IpV6 since you don't need at this point.

At this point you shall be able to VPN into your remote home/office.

2008-08-08T10:40:00.000-05:00

This is test I did in school when I was 10 years old (3d grade). See if your 10-11 old child will be able to make it. (Test for 7 year olds below this post)

Test 1.

Airplane needs to fly between 2 cities, distance between cities is 4150 km. First 2 hours airplane was flying 676 kmh, then 3 hours was flying with speed 760 kmh.

Question: How many km is left to fly?

Test 2.

Swimmer had swam 448 laps in first day. On second day he was swimming 2 hours less then first day and swam total of 336 laps.

Question: How many total hours swimmer was swimming on first day and on second day?

Test 3.

Your yard is 45 m in length and 24 m in width. What is perimeter of your yard?

Below answers from original paperwork filed in 1981

2008-08-08T10:24:00.000-05:00

This is test which I did at the end of 1st grade in Moscow School on May 17th, 1978 when I was 7 years old.

See if your 7-8 year old child can answer questions below.

Test 1.

There are 27 airplanes and 16 helicopters at airport. 18 aircrafts took off. Question: How many aircrafts left at airport?

Test 2.

64 + 23 = ?
14 - 8 = ?
48 + 5 = ?
53 + 27 = ?
9 + 6 = ?
90 - 26 = ?
Put <,>,= between 20-6 and 20-9

SSLScavenger available at codeplex

2008-08-04T20:17:00.001-05:00

If you work in environment with a lot of SSL sites and need to keep track of SSL certificates and their expiration dates you might find this utility usefull.
You can download it from http://www.codeplex.com/sslscavenger/.
It provides output in XML file if any of certificates are already expired or about to expire in 30 day period.

The best of freeware I find usefull

2008-06-05T16:30:00.003-05:00

Free remote access to home or office computers. All software and services listed below are completely free.

http://www.logmein.com/ has a free version of their product which will allow you to access any of your machines having LogMeIn installed form anywhere in the world. Current version supports popular features like multi monitor, clipboard sharing, chat.
If you need instantaneous and fast desktop sharing with your peer I'd recommend http://www.crossloop.com which is very easy to install and use. It's based on well established VNC based remote desktop access. Excellent choice if you need quickly connect to your peer.
http://ww.livemesh.com is in tech preview currently allows you to access your own remote computers and also supports file synchronisation both offline and online between devices.

Desktop heap issues and troubleshooting them

2008-04-04T13:05:00.008-05:00

I had weird behaviour on my desktop for the last couple of days. Windows would not open, windows which would open will miss items on them or tabs.
I immediately suspected desktop heap issues which I run into before (becouse I have 3 monitors). I checked registry key (HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows), SharedSection=1024,3072,512. Last number in that value is recommended value Microsoft suggests if there are any issues with desktop heap performance.
Apparently I had already value set and was still experiencing issues.
Next stop is to actually verify this issue is in fact relevant to heap exhaustion.
You need to download and install tool called dheapmon from location below
Dheapmon (Desktop Heap Monitor)
http://www.microsoft.com/downloads/details.aspx?familyid=5CFC9B74-97AA-4510-B4B9-B2DC98C8ED8B&displaylang=en
Results of running that tool on my desktop are below


C:\kktools\dheapmon8.1\x86>dheapmon -s
Desktop Heap Information Monitor Tool (Version 8.1.2925.0)
Copyright (c) Microsoft Corporation.  All rights reserved.
-------------------------------------------------------------
  Session ID:    0 Total Desktop: (  5824 KB -    8 desktops)

  WinStation\Desktop            Heap Size(KB)    Used Rate(%)
-------------------------------------------------------------
  WinSta0\Default                    3072             93.3
  WinSta0\Disconnect                   64              4.5
  WinSta0\Winlogon                    128              9.4
  Service-0x0-3e7$\Default            512             24.8
  Service-0x0-3e4$\Default            512              7.4
  Service-0x0-3e5$\Default            512              2.4
  SAWinSta\SADesktop                  512              0.5
  msswindowstation\mssrestricteddesk    512              0.5
-------------------------------------------------------------

It's clear that WinSta0\Default is causing all the pr0blems. This desktop is actual desktop I'm using so one of processes running under my account caused heap exhaustion. I did not found a way to find what is on that heap, so had to go through easiest way. Open task manager, and sort processes owned by my user account and start killing them one by one while refreshing dheapmon output while doing so.

After 3 proccesses which were killed which did not yeld any results, I killed "Microsoft Office Communicator 2005" which dropped heap use from 90% to 40%.


C:\kktools\dheapmon8.1\x86>dheapmon -v -w WinSta0 -d Default
Desktop Heap Information Monitor Tool (Version 8.1.2925.0)
Copyright (c) Microsoft Corporation.  All rights reserved.
-------------------------------------------------------------

  Windowstation: (WinSta0) SessionID: 0
  Desktop: (Default) Addr: b96d0000

  Desktop Heap         3145728 (0x  300000) Bytes
  Committed            3141632 (0x  2ff000) Bytes
  Uncommitted             4096 (0x    1000) Bytes
  Allocated            1283816 (0x  1396e8) Bytes
  Total Freed          1857816 (0x  1c5918) Bytes
  Unused               1861912 (0x  1c6918) Bytes
  Used Rate                    (      40.8) %

I'm yet to reproduce this issue again to raise question with MS

2008-04-02T08:27:00.000-05:00

Stackoverflow exception analysis in crashed .NET 2.0 application

This is second crash analysis I had to perform to find out what is causing web process to crash. Prerequisite as in original crash analysis are Windows Debugging Tools installed (WinDbg), point symbols locations to Microsoft download servers and load SOS for .NET 2.0 to see clr related objects. Commands entered by me and in bold letters, debugger output is in gray, areas of interest are in red. Detailed information about prerequisites are in previous blog in 2007.

Microsoft (R) Windows Debugger Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.
Extracted C:DOCUME~1GSUVAL~1LOCALS~1Temp10000137c0_mdmpmem.hdmp from
C:WINDOWSPCHealthERRORREPQSIGNOFF47D95C22.cab

Loading Dump File [C:DOCUME~1GSUVAL~1LOCALS~1Temp10000137c0_mdmpmem.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Symbol search path is: SRV*c:symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Version 3790 (Service Pack 1) MP (4 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Debug session time: Tue Apr 1 21:09:53.000 2008 (GMT-5)
System Uptime: not available
Process Uptime: 0 days 5:06:56.000
.....................................................................................................................
The call to LoadLibrary(exts) failed, Win32 error 0n2
"The system cannot find the file specified."
Please check your debugger configuration and/or network access.
The call to LoadLibrary(ntsdexts) failed, Win32 error 0n2
"The system cannot find the file specified."
Please check your debugger configuration and/or network access.
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(ab4.11d0): Unknown exception - code e053534f (first/second chance not available)
eax=00a80000 ebx=77e670da ecx=00000019 edx=0000001c esi=00000154 edi=00000000
eip=7c82ed54 esp=01af3de8 ebp=01af3e58 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200297
ntdll!KiFastSystemCallRet:
7c82ed54 c3 ret

Load sos.dll from your .NET 2.0 directory with command below to enable .NET managed code access

0:014> .load C:WINDOWSMicrosoft.NETFrameworkv2.0.50727sos.dll
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------

See what managed exception you got (print exception), we can see that we got Exception of type StackOverflowException

0:014> !pe
Exception object: 142d00a4
Exception type: System.StackOverflowException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 800703e9

Now we need to see what we got on CLR stack by executing command below. You can see that there is infinite loop of cache being inserted and CacheitemRemovedCallback being called on that event. This documented issue (http://blogs.msdn.com/yangxind/archive/2005/12/20/506151.aspx) and solution is to use Cache.Add instead of Cache.Insert statement for .NET 2.0 applications.

0:014> !clrstack
OS Thread Id: 0x11d0 (14)
ESP EIP
01af4b24 7c82ed54 [FaultingExceptionFrame: 01af4b24]
01af5104 7c82ed54 [HelperMethodFrame: 01af5104]
01af5170 7a608d56 System.Net.Sockets.Socket.MultipleSend(System.Net.BufferOffsetSize[], System.Net.Sockets.SocketFlags)
01af51b4 7a5d43b7 System.Net.Sockets.NetworkStream.MultipleWrite(System.Net.BufferOffsetSize[])
01af51f4 7a5af41d System.Net.Connection.Write(System.Net.ScatterGatherBuffers)
01af51fc 7a5b1433 System.Net.ConnectStream.ResubmitWrite(System.Net.ConnectStream, Boolean)
01af525c 7a57f835 System.Net.HttpWebRequest.EndWriteHeaders_Part2()
01af528c 7a57e002 System.Net.HttpWebRequest.SetRequestContinue(System.Net.CoreResponseData)
01af52a8 7a5af3be System.Net.Connection.ReadComplete(Int32, System.Net.WebExceptionStatus)
01af52e8 7a5af008 System.Net.Connection.SyncRead(System.Net.HttpWebRequest, Boolean, Boolean)
01af533c 7a5aede2 System.Net.Connection.PollAndRead(System.Net.HttpWebRequest, Boolean)
01af5350 7a5b412b System.Net.ConnectStream.PollAndRead(Boolean)
01af535c 7a57f697 System.Net.HttpWebRequest.EndWriteHeaders(Boolean)
01af538c 7a57dee2 System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus, System.Net.ConnectStream, Boolean)
01af5398 7a5b40f6 System.Net.ConnectStream.WriteHeaders(Boolean)
01af53e4 7a57f4da System.Net.HttpWebRequest.EndSubmitRequest()
01af5410 7a57c78b System.Net.HttpWebRequest.CheckDeferredCallDone(System.Net.ConnectStream)
01af5420 7a57ca5e System.Net.HttpWebRequest.GetResponse()
01af5468 6990bf7d System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(System.Net.WebRequest)
01af5498 6990c915 System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(System.Net.WebRequest)
01af549c 69919eb1 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[])
01af54e0 03745370 eServiceWS.eServiceLiteWS.getListofServiceRequestAreaSubArea()
01af54ec 037450b4 Controller.GetLOVList(LOVTypes)
01af5530 01d3bb07 StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5584 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af55c0 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af55fc 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af56cc 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af56f0 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af574c 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af5784 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af57d8 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5814 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5850 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5920 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5944 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af59a0 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af59d8 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5a2c 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5a68 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5aa4 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5b74 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5b98 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af5bf4 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af5c2c 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5c80 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5cbc 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5cf8 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5dc8 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af5dec 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af5e48 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af5e80 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af5ed4 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af5f10 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af5f4c 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af601c 65fa82ea System.Web.Caching.CacheMultiple.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01af6040 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01af609c 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01af60d4 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01af6128 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01af6164 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01af61a0 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean,
01b1f570 65fa8e2a System.Web.Caching.CacheInternal.DoInsert(Boolean, System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback, Boolean)
01b1f5cc 6601bc92 System.Web.Caching.Cache.Insert(System.String, System.Object, System.Web.Caching.CacheDependency, System.DateTime, System.TimeSpan, System.Web.Caching.CacheItemPriority, System.Web.Caching.CacheItemRemovedCallback)
01b1f604 01d3bb8d StaticCache.onAreaSubAreaLOVRemove(System.String, System.Object, System.Web.Caching.CacheItemRemovedReason)
01b1f658 6601e362 System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(System.Web.Caching.CacheItemRemovedCallback, System.Web.Caching.CacheItemRemovedReason)
01b1f694 6601e60d System.Web.Caching.CacheEntry.Close(System.Web.Caching.CacheItemRemovedReason)
01b1f6d0 6630239c System.Web.Caching.CacheSingle.UpdateCache(System.Web.Caching.CacheKey, System.Web.Caching.CacheEntry, Boolean, System.Web.Caching.CacheItemRemovedReason, System.Object ByRef)
01b1f7a0 6601c398 System.Web.Caching.CacheInternal.DoRemove(System.Web.Caching.CacheKey, System.Web.Caching.CacheItemRemovedReason)
01b1f7ac 66023ad2 System.Web.Caching.ExpiresBucket.FlushExpiredItems(System.DateTime, Boolean)
01b1f820 66023c6f System.Web.Caching.CacheExpires.FlushExpiredItems(Boolean, Boolean)
01b1f868 66023d65 System.Web.Caching.CacheExpires.TimerCallback(System.Object)
01b1f86c 793d9c1a System.Threading._TimerCallback.TimerCallback_Context(System.Object)
01b1f870 793683dd System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
01b1f888 793d9c7f System.Threading._TimerCallback.PerformTimerCallback(System.Object)
01b1fa18 79e88f63 [GCFrame: 01b1fa18]
01b1fb5c 79e88f63 [ContextTransitionFrame: 01b1fb5c]

2007-01-30T08:40:00.000-06:00

ASP.NET debugging using memory crash analysis

I was given a task to find out why ASP.NET process was crashing several times an hour on production machines. The only information I had was couple of entries in Event Log.

Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1000
Date: 10/13/2006
Time: 1:02:32 PM
User: N/A
Computer: DALPRDSIINT6
Description:
aspnet_wp.exe (PID: 16620) stopped unexpectedly.

Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 5000
Date: 10/13/2006
Time: 1:02:28 PM
User: N/A
Computer: DALPRDSIINT6
Description:
EventType clr20r3, P1 aspnet_wp.exe, P2 2.0.50727.42, P3 4333aece, P4 system.data, P5 2.0.0.0, P6 4333aea2, P7 1946, P8 25, P9 system.invalidoperationexception, P10 NIL.

Here is the steps I took based on this article

Install Debugging tools for Windows
Locate you crash dumps in following location
Start Windbg program and modify symbol path to point to Microsoft servers
Open crash dump file
Results of operation with crash dump are below, stuff in red are commands and stuff in blue are relevant pieces

Extracted C:DOCUME~1A-GREG~1.SUVLOCALS~1Temp1700003cac0_mdmpmem.hdmp from
C:WINNTPCHEALTHERRORREPQSIGNOFFE59B571D.cab
Loading Dump File [C:DOCUME~1A-GREG~1.SUVLOCALS~1Temp1700003cac0_mdmpmem.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Windows 2000 Version 2195 (Service Pack 4) UP Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Debug session time: Wed Sep 27 15:46:57.000 2006 (GMT-5)
System Uptime: not available
Process Uptime: 0 days 0:03:06.000
Symbol search path is: srv*c:symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
..................................................................................................
------------------------------------------------------------
sos.dll needs a full memory dump for complete functionality.
You can create one with .dump /ma <filename>
------------------------------------------------------------
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1d84.3eec): CLR exception - code e0434f4d (first/second chance not available)
eax=06030000 ebx=02dee60c ecx=00001000 edx=00000000 esi=77f88f08 edi=000007ac
eip=77f88f13 esp=02dee5f0 ebp=02dee614 iopl=0 nv up ei ng nz ac pe cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297
NTDLL!ZwWaitForSingleObject+0xb:
77f88f13 c20c00 ret 0Ch
Dumping list of threads including thread where exception occurred (in our case thread no 6)
0:006> !threads
ThreadCount: 9
UnstartedThread: 0
BackgroundThread: 9
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
1 1 3ad8 00164c48 8808220 Enabled 00000000:00000000 0015c550 0 Ukn (Threadpool Completion Port)
6 2 3eec 00170710 b220 Enabled 00000000:00000000 05227d48 1 MTA (Finalizer) System.InvalidOperationException (01003f2c)
7 3 3294 001866e8 80a220 Enabled 00000000:00000000 0015c550 0 MTA (Threadpool Completion Port)
10 4 3d48 00198c68 1220 Enabled 00000000:00000000 0015c550 0 Ukn
11 5 246c 001e27d8 880b220 Enabled 00000000:00000000 0015c550 0 MTA (Threadpool Completion Port)
13 6 37c 00200c08 180b220 Enabled 00000000:00000000 0015c550 0 MTA (Threadpool Worker)
9 8 8fc 0023a8e0 220 Enabled 00000000:00000000 0015c550 0 STA
15 9 7dc 002330b8 200b220 Enabled 00000000:00000000 002124c0 1 MTA
16 7 ae0 00233478 880b220 Enabled 00000000:00000000 0015c550 0 MTA (Threadpool Completion Port)
Find information about exception based on address above

0:006> !printexception 01003f2c
Exception object: 01003f2c
Exception type: System.InvalidOperationException
Message: Internal .Net Framework Data Provider error 1.
InnerException: <none>
StackTrace (generated):
SP IP Function
02DEF8D4 653976BF System.Data.ProviderBase.DbConnectionInternal.PrePush(System.Object)
02DEF8E0 652B29CD System.Data.ProviderBase.DbConnectionPool.PutObject(System.Data.ProviderBase.DbConnectionInternal, System.Object)
02DEF920 65277FC6 System.Data.ProviderBase.DbConnectionInternal.CloseConnection(System.Data.Common.DbConnection, System.Data.ProviderBase.DbConnectionFactory)
02DEF950 652CD4BA System.Data.SqlClient.SqlConnection.Close()
02DEF988 05E5299E RNTransactionLogWS.RNTransactionLog.Finalize()
StackTraceString: <none>
HResult: 80131509
Our InvalidOperationException happened in Finalize() method of RNTransacationLog. We need to dump contents of that assembly, first we using IP to MD command to find method descriptor

0:006> !ip2md 0x05E5299E
MethodDesc: 05c78bb8
Method Name: RNTransactionLogWS.RNTransactionLog.Finalize()
Class: 05e3544c
MethodTable: 05c78c5c
mdToken: 06000035
Module: 05c75534
IsJitted: yes
m_CodeOrIL: 05e52960
Dump method table to find which DLL code compiled to
0:006> !dumpmt 05c78c5c
EEClass: 05e3544c
Module: 05c75534
Name: RNTransactionLogWS.RNTransactionLog
mdToken: 02000008 (C:WINNTMicrosoft.NETFrameworkv2.0.50727Temporary ASP.NET Filesiis_rnlogging8f383ed1f56b55d8assemblydl36231f022a01f4daa_e7a8c501RNTransactionLogWS.DLL)
BaseSize: 0x18
ComponentSize: 0x0
Number of IFaces in IFaceMap: 0
Slots in VTable: 23
Find loading address by using command lmv<assemblyname>
0:006> lmv mRNTransactionLogWS
start end module name
05e40000 05e50000 RNTransactionLogWS (deferred)
Image path: C:WINNTMicrosoft.NETFrameworkv2.0.50727Temporary ASP.NET Filesiis_rnlogging8f383ed1f56b55d8assemblydl36231f022a01f4daa_e7a8c501RNTransactionLogWS.DLL
Image name: RNTransactionLogWS.DLL
Has CLR image header, track-debug-data flag not set
Timestamp: Mon Dec 05 12:24:57 2005 (439485F9)
CheckSum: 00000000
ImageSize: 00010000
File version: 1.0.2165.22349
Product version: 1.0.2165.22349
File flags: 0 (Mask 3F)
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0
CompanyName:
ProductName:
InternalName: RNTransactionLogWS.dll
OriginalFilename: RNTransactionLogWS.dll
  ProductVersion: 1.0.2165.22349
FileVersion: 1.0.2165.22349
FileDescription:
LegalCopyright:
LegalTrademarks:
Comments:
Now we can extract assembly from memory dump
0:006> !savemodule 05e40000 c:a.dll
3 sections in file
section 0 - VA=2000, VASize=8af4, FileAddr=1000, FileSize=9000
section 1 - VA=c000, VASize=390, FileAddr=a000, FileSize=1000
section 2 - VA=e000, VASize=c, FileAddr=b000, FileSize=1000

The last is finding offending code in assembly. Ildasm.exe can be used or Lutz Roeder's reflector from http://www.aisto.com/roeder/dotnet/. Output of reflector shown below highlighting offending method where exception happened (Finalize())

We can see that developer tried to close Connection object within Finalize() method which is explicitly stated in MSDN as prohibited. Here is part of documentation

Caution
Do not call Close or Dispose on a Connection, a DataReader, or any other managed object in the Finalize method of your class. In a finalizer, you should only release unmanaged resources that your class owns directly. If your class does not own any unmanaged resources, do not include a Finalize method in your class definition. For more information, see Garbage Collection.

Bingo! We know how, when and why application was crashing.