397 liens privés
A few weeks ago I faced an interesting problem trying to analyze a memory consumption in my Java application (Spring Boot + Infinispan) running under Docker. The Xmx parameter was set to 256m, but the Docker monitoring tool displayed almost two times more used memory. Below we will try to understand the reasons of such a strange behavior and find out how much memory the app consumed in fact.
Je ne connaissais pas le Native Memory Tracker du jdk8
TL;DR: The JVM by default exports statistics by mmap-ing a file in /tmp (hsperfdata). On Linux, modifying a memory mapped file can block until disk I/O completes, which can be hundreds of milliseconds. Since the JVM modifies these statistics during garbage collection and safepoints, this causes pauses that are hundreds of milliseconds long. To reduce worst-case pause latencies, add the -XX:+PerfDisableSharedMem JVM flag to disable this feature. This will break tools that read this file, like jstat.
Bien penser à limiter et dimensionner le ramdisk en proportion de la ram.
"""
- Don’t hotdeploy/redeploy/migrate your java services in production at runtime
- Do have a very strong focus on your delivery pipeline/automation/testing to quickly make changes to your system
"""
Autre lecture conseillée: https://medium.com/fabric8-io/the-decline-of-java-application-servers-when-using-docker-containers-edbe032e1f30
Petite formule saltstack pour faire utiliser les 3/4 de la RAM à la JVM:
"""
Xmx: {{ (((grains.mem_total|int) 3) / 4)|round|int }}M
Xms: {{ (((grains.mem_total|int) 3) / 4)|round|int }}M
"""
J'ai positionné cela dans les pillars.
C'est inspiré de https://mywushublog.com/2013/09/postgresql-salt-state/ (https://jeekajoo.eu/links/?crzrTg)
Cela évite d'avoir à se palucher un tuning manuel pour des machines dont les capacités mémoire sont différentes. Exemple sur ec2:
- m3.2xlarge = 30 G
- c3.2xlarge = 15 G
- c3.xlarge = 7.5 G
- m3.medium = 3.75 G
- ....
Pour la partie serveur, j'utilise jolokia dont le jar est chargé en tant qu'agent de la jvm, de cette façon:
-javaagent:/var/lib/tomcat8/lib/jolokia-jvm-agent.jar=port=7777,host=localhost,agentContext=/jolokia
Pour la partie cliente, j'utilise la lib python 'pyjolokia': https://github.com/cwood/pyjolokia . Ce client me permet de faire un monitoring de la JVM en python. Ce qui est appréciable.
Voici ma méthode pour lister les mbeans, en mode intéractif via une console python:
initialisation
from pyjolokia import Jolokia
j4p = Jolokia('http://localhost:7777/jolokia/')
lister les mbeans dans java.lang:*
j4p.request(type = 'search', mbean='java.lang:')
{u'status': 200, u'timestamp': 1429886584, u'request': {u'type': u'search', u'mbean': u'java.lang:'}, u'value': [u'java.lang:name=Metaspace,type=MemoryPool', u'java.lang:type=Runtime', u'java.lang:type=Threading', u'java.lang:type=OperatingSystem', u'java.lang:name=Code Cache,type=MemoryPool', u'java.lang:type=Compilation', u'java.lang:name=G1 Young Generation,type=GarbageCollector', u'java.lang:name=CodeCacheManager,type=MemoryManager', u'java.lang:name=Compressed Class Space,type=MemoryPool', u'java.lang:type=Memory', u'java.lang:name=G1 Eden Space,type=MemoryPool', u'java.lang:name=G1 Old Gen,type=MemoryPool', u'java.lang:name=G1 Old Generation,type=GarbageCollector', u'java.lang:type=ClassLoading', u'java.lang:name=Metaspace Manager,type=MemoryManager', u'java.lang:name=G1 Survivor Space,type=MemoryPool']}
lister les attributs d'un mbean en particulier, par exemple java.lang:name=G1 Old Generation,type=GarbageCollector
j4p.request(type = 'list', path='java.lang/type=GarbageCollector,name=G1 Old Generation')
{u'status': 200, u'timestamp': 1429886821, u'request': {u'path': u'java.lang/type=GarbageCollector,name=G1 Old Generation', u'type': u'list'}, u'value': {u'attr': {u'CollectionTime': {u'rw': False, u'type': u'long', u'desc': u'CollectionTime'}, u'LastGcInfo': {u'rw': False, u'type': u'javax.management.openmbean.CompositeData', u'desc': u'LastGcInfo'}, u'Name': {u'rw': False, u'type': u'java.lang.String', u'desc': u'Name'}, u'ObjectName': {u'rw': False, u'type': u'javax.management.ObjectName', u'desc': u'ObjectName'}, u'Valid': {u'rw': False, u'type': u'boolean', u'desc': u'Valid'}, u'MemoryPoolNames': {u'rw': False, u'type': u'[Ljava.lang.String;', u'desc': u'MemoryPoolNames'}, u'CollectionCount': {u'rw': False, u'type': u'long', u'desc': u'CollectionCount'}}, u'desc': u'Information on the management interface of the MBean'}}
Voila voila, c'est un peu austère comme méthode mais ça marche.
la JVM n'aura plus de secret pour vous.
