On a Windows Server with GUI, you can use the traditional method of setting the display resolution but as there's no Control Panel nor Settings app on Windows Server Core, the only way is through Powershell with the Set-DisplayResolution cmdlet.

Set-DisplayResolution -Width 1024 -Height 768

The cmdlet above sets the resolution to 1024x768. It will prompt you to accept the change, if you don't want that, append -Force parameter.

To get the current display resolution, you can use the Get-DisplayResolution cmdlet.