Pig : Container is running beyond physical memory
Pig : Container is running beyond physical memory limits using oozie
Here is a good answer from: http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits
Looks like the default for YARN container size on that cluster (1 GB of RAM) is way too low for your job.
But it’s not clear whether the YARN error shown relates to the Shell action (running grunt) or to a child MapReduce execution.
Plan A - Assuming that it’s the child MapReduce execution that requires more RAM, on top of the grunt script just add
|
1 2 3 |
set mapreduce.map.memory.mb 5120 set mapreduce.reduce.memory.mb 5120 |
Reference: Pig documentation, YARN documentation
Plan B - If it is actually the Shell action that requires more RAM, add some configuration in workflow.xml
|
1 2 3 4 5 6 7 8 9 10 11 12 |
<action name="PiggyBack"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>oozie.launcher.mapreduce.map.memory.mb</name> <value>5120</value> </property> </configuration> |











