There are plenty of tools available, free and otherwise, to provide access to SFTP and SCP servers, with more or less features. Unfortunately, there aren't a whole lot of extensions out there that properly support SFTP/SCP through Windows Explorer itself, providing a somewhat seamless integration in Windows itself. Although there is the win-sshfs project, but this requires installation of several additional prerequisited software components, making this a less attractive solution. Luckily, there's also Swish.

Swish provides SFTP access from inside Windows Explorer. As it installs as an extension, it has no additional requirements, nor does it need to install some driver or library to work. It also has integration with Pageant so you can use SSH keys for authentication. Their official website holds the download and all documentation you need: http://www.swish-sftp.org/.

Because Swish is still in development, there are still a few features missing. The most important is probably the ability to do in-place opening and editing of files, but for now, this can be worked around by copying the file locally, editing it, then put it back on the server. Also, symlink support is not entirely working.