If the Windows Internal Database, used by WSUS, and which is actually an embedded SQL Server, is eating all of your memory, you can limit its memory usage the same way you would with a real SQL Server.

Open an elevated command prompt and launch the SQL prompt:

For WSUS4 (on Server 2012):

osql -E -S \.pipeMicrosoft##WIDtsqlquery

For WSUS3 (on Server 2008, 2008R2 and SBS 2008/2011)

osql -E -S \.pipemssql$microsoft##sseesqlquery

This will log you in with the current credentials (which is why it has to be an elevated command prompt). The SQL prompt (1>) is shown.

Enter the following commands to initiate advanced configuration:

exec sp_configure 'show advanced option', '1';
reconfigure;

To view the current setting for maximum usable server memory:

exec sp_configure;
go

Look for the line max server memory. By default, there's no limitation (i.e. it's set "really" high). Its value is displayed in MB.

To reconfigure (e.g. for 2GB):

exec sp_configure 'max server memory', 2048;
reconfigure with override;
go

quit

Don't forget to restart the Windows Internal Database service.