Found this blog post shortly after our meeting: http://blog.bruchez.name/2013/02/mapmap-vs-mapmapvalues.html
I learned that by using mapValues, I'm actually continually constructing MappedValues instances on top of the original map. It seems like this is at least part of the memory leak that @jasongilanfarr was referring to. It's not a "leak" per-se, but we believe we continue to layer MappedValues after another on top of the original map.
When running the following bash script:
#!/bin/bash for i in {1..300}; do http POST :8080/v2/apps <<EOF { "id" : "/$(($i % 17))/$(($i % 13))/$(($i % 11))/$(($i % 7))/simple-app-$i", "cmd": "sleep 1000", "instances": 1, "cpus": 0.01, "mem": 1, "disk": 0 } EOF done
- On master, I was only able to get to about 150 before it basically stopped moving, and JMC claimed that I had about 1.2 million instances of Group being constructed already.
- On master + RootGroup reverted, I got to 300, and had about 170,000 instances.
- D320 + with this patch, I got to 300 and had about 39,000 instances.