It's more than that, because generally performance is also much worse if you're using a shit ton of memory. That's because CPUs are bottlenecked by cache. So more memory means cache has to be flushed more often, and there will be more misses, which can greatly impact performance.