Yes, the sonar findbugs executor can use a lot of RAM if the analyzed code base is large.
You will have to increase not only the heap size available to the JVM, but also "MaxPermSize" and "ReservedCodeCacheSize" (at least for the Java Hotspot VM). The JVM options must be set on the JVM which is actually executing the FindBugs sonar executor. So probably not the Jenkins VM, but the one that Jenkins starts for the job.
Here's an example for a 64bit system:
-Xmx3062m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=128m
On a 32bit system, which I am assuming you are using, you would go for lower values, such as:
-Xmx1536m -XX:MaxPermSize=384m -XX:ReservedCodeCacheSize=128m
Also, if Jenkins won't accept the JVM options, you can try running sonar using the sonar ant task from Jenkins. (That's what I am doing, and I can set as much memory as I like.)
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…