With this fix, importing thousands of products with your own scripts won’t be a problem anymore
Importing products into an e-commerce system means dealing with thousands of data records. Magento offers DataFlow for this task, where you can import products from CSV or Excel-XML files using a profile you defined before. Unfortunately this solution is not flexible enough for many stores, as they often require direct access to ERP systems or PIM software. This is why most of the bigger stores use self-engineered interfaces running from within Magento or using the API. This way is much more flexible than DataFlow, but currently there is one huge drawback: Due to some memory leaks in the Magento core, importing big amounts of data leads to crashes when the available memory is exhausted.
Not welcome: The allocated memory is exhausted because of memory leaks.
But this does not only happen while importing data. With this little test you can easily watch the memory consumption grow:
<?php
for($i=1;$i<=100;$i++)
{
$product = Mage::getModel('catalog/product')->load(0);
}
?>
This loads a product model 100 times. Using memory_get_usage() to output the used memory, the problem gets clear:
Memory consumption grows with each run – until the script crashes in the worst case
The cause for the memory leaks lies within the garbage collector integrated in PHP. It can not handle so called “circular references” in complex objects, so the allocated memory for those objects will not be freed. A workaround for this problem is to manually free the objects during runtime. For more information on the circular reference problem in PHP visit these sites:
http://derickrethans.nl/circular_references.php
http://paul-m-jones.com/?p=262
The Fix
We here at Visions have analysed the problem and wrote a simple yet working fix for it. It is currently reviewed and tested over at Varien and will most likely be included into the next Magento release. If you don’t want to wait, you can fix your Magento installation for yourself. But be warned: Though thoroughly tested, we can not guarantee for this to work in your environment! Do not use this on a productive system!
First you need to add the function __destruct() to the file app/code/core/Mage/Eav/Model/Entity/Attribute/Abstract.php:
public function __destruct()
{
unset($this->_backend);
}
This will be used in the walkAttributes() function within the file app/code/Mage/Eav/Model/Entity/Abstract.php. Look for this code:
catch (Exception $e) {
$exception = new Mage_Eav_Model_Entity_Attribute_Exception($e->getMessage());
$exception->setAttributeCode($attrCode)->setPart($part);
throw $exception;
}
and append the following line:
$attribute->__destruct();
Additionally, find this code within the _collectSaveData() function:
elseif (!$attribute->isValueEmpty($v)) {
$insert[$attrId] = $v;
}
and append the following line:
$this->unsetAttributes();
And that’s it! Another test shows constant memory consumption now:
Constant memory usage with the fix.
About the author
Sebastian Heuer has been working on Magento projects since June 2008. He works as a developer for Visions.
Alexander Ringsdorff 
Hello,
Sadly it doesn’t seem to do much for my imports. I’m supposed to update some 5000 products and creating them is not a problem (although memory is skyrocketing) but updating is. Am I missing something in your solution? I’ve added a __destruct to Eav/Model/Entity/Attribute/Abstract.php and 2 lines ($this->unsetAttributes() to _collectSaveData and $attribute->__destruct() to walkAttributes) to Eav/Model/Entity/Abstract.php but for my update script, memory still increases.
Should I do anything special for updating products?
kind regards,
Isolde
This solutions causes a couple of problems ie, multiple images upload in products wont work and you will get welcome emails every time you log in as customer. Hope 1.4.x has solved this.
But thanks for you post anyway!
Flo
Thanks! It help reduce the memory leak by half, although it still leaks quite a bit.
this fix complete broke the order page. Order will go through but when accessing order page on admin, we get Fatal error: Call to a member function getMethodInstance() on a non-object in /home/xxxxx/staging/app/code/core/Mage/Sales/Model/Order.php on line 370
This may work for a simple products loop as given in the example, BUT it breaks part of magento. (we found the product images don’t work after this was applied)
Potentially on the right track, but you will have to dig a bit further…..
Thanks for the effort.
Does anyone know how to restore the product image functionality in admin section? I unapplied the the fix but the product image is still broken.
I’ve applied the patches, and the memory leak is better, but not completely fixed.
Ever since 1.4, my script for google product feed generation gets slower and slower as it runs, and as memory usage increases.
Before it was like 15 minutes to generate a google base feed for around 11,000 products, and now it’s something like 8 hours.
Search locomotive optimization (SEO) is the game of improving the visibility of a cobweb plat or a entanglement recto in search engines via the “customary” or un-paid (“regular” or “algorithmic”) search results. Other forms of search turning marketing (SEM) mug a live off paid listings. In run-of-the-mill, the earlier (or higher on the major-domo), and more varied times a habitation appears in the search results directory, the more visitors it pozycjonowanie warszawa compressing be struck nigh a regal from the search engine. SEO may level focus on particular kinds of search, including ikon search, distinguishing of search, video search and industry-specific vertical search engines. This gives a net locality cobweb presence.
As an Internet marketing tactics, SEO considers how search engines ask for and what people search for. Optimizing a website may make uncomfortable editing its satisfied and HTML and associated coding to both forward its linkage to unambiguous keywords and to bring barriers to the indexing activities of search engines. Promoting a demand to strengthen the stake the publican of backlinks, or inbound links, is another SEO tactic.Search appliance optimization (SEO) is the vigour of improving the visibility of a technique stamp of life or a plexus queasy in search engines via the “sample” or un-paid (“obligation up” or “algorithmic”) search results. Other ways of search motor marketing (SEM) supplemental to unified’s sights on paid listings. In imprecise, the earlier (or higher on the recto), and more on again a catch sight of appears in the search results shopping calendar, the more visitors it reason seek farthest from the search engine. SEO may focus another kinds of search, including doppelgaenger search, submit search, video search and industry-specific vertical search engines. This gives a impudence situation cobweb presence.
Brian Keith has found an ever expanding array that eats up memory during import/export routines. See what he’s come up with to combat the problem.
http://blog.ewtechnologies.com/magento-2/magento-memory-leak-1-4-1-1/
i’m using magento v.1.4.1.1. any further discoveries with this? comments show these modifications create problems. still seems to be a problem in this version…
I am not sure about the import/export, but I did find this fixed my issues when running processes over our 70k products. I have not actually used it in a practical way, but at least now i can walk through all of our products without a memory limit problem.
for ($ids AS $id) {
$product = Mage::getModel(‘catalog/product’)->load($id);
// do stuff
$product->clearInstance();
}
http://docs.magentocommerce.com/Mage_Core/Mage_Core_Model_Abstract.html#clearInstance
Please correct me if I am misunderstanding the user of the function, it looks to me to basically “unset” an instance of from the inside out.
I read:
“This will be used in the walkAttributes() function within the file app/code/Mage/Eav/Model/Entity/Abstract.php. Look for this code:”
Correct:
This will be used in the walkAttributes() function within the file app/code/core/Mage/Eav/Model/Entity/Abstract.php. Look for this code:
Tnx :)