Having similar problem with any games. Tried using stable release (1.6.0) and latest dev, the problem still occurs. For me it happened when choose adapter to nvidia, but change to "Intel HD Graphics", the game run normal with "lag" / fps drops sometimes (Direct3D 11 (Hardware) renderer). When choose to nvidia adapter, the game still playable but no visual shows (only black screen), and the log just repeated "GSDX OUT OF MEMORY" or "GSdx: Memory allocation error"
Using 4gb patch, as someone suggested it, had no effect, and "GSDX OUT OF MEMORY" still shows.
This is my setting and log when playing suikoden IV