tag:blogger.com,1999:blog-33578521.post2001003300863628303..comments2023-06-11T03:23:03.429-07:00Comments on mushkevych: Recovering from OutOfMemoryError in Hadoop mapreduceBohdan Mushkevychhttp://www.blogger.com/profile/09852569293214404261noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-33578521.post-32205392651613818682012-04-19T16:55:28.056-07:002012-04-19T16:55:28.056-07:00Good questions. Let me provide answers:
- in cases...Good questions. Let me provide answers:<br />- in cases with user-driven content among hundreds of thousands accounts, you might expect few that simply fall short from your predictions and expectations; <br />- it is not always money-wise to pay for additional GB of RAM if you need them to compute >0.01% of (usually fake) data<br />- in this particular case mappers were set to 512MB, while reducers to 768 MB; most of the processing fitted well into 20-30 MB<br />- in reality try{} block should be build with understanding of JVM RAM Allocation mechanics.<br />From [A] we can find out that JVM does computation in advance before allocating the object. <br />Since Hadoop Table mappers and reducers are single threaded, we can expect that memory allocation is located in try() block. <br />By keeping all references inside try{} block and by trying to allocate large chunk of RAM we secure ourself from running into bad OutOfMemory situations, when Heap is full. <br /><br />[A] OOME discussion on stackoverflow: http://stackoverflow.com/questions/9261705/what-happens-when-theres-insufficient-memory-to-throw-an-outofmemoryerrorBohdan Mushkevychhttps://www.blogger.com/profile/09852569293214404261noreply@blogger.comtag:blogger.com,1999:blog-33578521.post-78521031977207320692012-03-30T07:27:52.912-07:002012-03-30T07:27:52.912-07:00How can you be sure by the time to you get to catc...How can you be sure by the time to you get to catch block jvm is in such a state that anything can be done? e.g. run gc manually etc<br /><br />You're making good point about requirements that are user data driven but still it seems like you're fighting your system resources threshold. Try to figure out peak load and just double the resources to handle it. No always possible but with clouds and dynamic quotas it can be configured for less.<br /><br />What amount of data on ave are we talking about in your case?RPhttps://www.blogger.com/profile/03914962807074429088noreply@blogger.comtag:blogger.com,1999:blog-33578521.post-49462790848651538022012-03-28T21:02:23.473-07:002012-03-28T21:02:23.473-07:001. It is easier to avoid OoME then to recover.
2. ...1. It is easier to avoid OoME then to recover.<br />2. Sometimes it is cheaper to buy additional 16G then to develop & test recovery code.Unknownhttps://www.blogger.com/profile/00425417467164570059noreply@blogger.com