If the Windows Internal Database, used by WSUS, and which is actually an embedded SQL Server, is eating all of your memory, you can limit its memory usage the same way you would with a real SQL Server.

Open an elevated command prompt and launch the SQL prompt:

For WSUS4 (on Server 2012):

osql -E -S \.pipeMicrosoft##WIDtsqlquery

For WSUS3 (on Server 2008, 2008R2 and SBS 2008/2011)

osql -E -S \.pipemssql$microsoft##sseesqlquery

This will log you in with the current credentials (which is why it has to be an elevated command prompt). The SQL prompt (1>) is shown.

Enter the following commands to initiate advanced configuration:

exec sp_configure 'show advanced option', '1';reconfigure;  ...