» Magento Commerce – Layered Navigation Cache Error – Part Deux | Chris Mckee Designs, by Chris McKee, who'd of guessed

- Magento Commerce – Layered Navigation Cache Error – Part Deux
  March 19, 2009
- Solution 2 – Running the Layered Navigation Cache in bits from PHP-CLI
  
  Part Three Solves a lot (GO BADGER)
  
  So not everyone has half a gig of ram to play with, that’s a git. Funnily though, I don’t have PS on the server I’m developing Magento on (why, oh god why?) so I cant build a shareable version of a web page script yet (till I have time to build a dev server).
  
  Anyhow I’ve built you a nice script that will process 40 (you can change this) at once, and store its progress in a text file, this allows us to come back later, but more importantly terminating the script after a certain number allows Zend to clean up after mage and means we can then just run the script again and allow it to continue with the next batch. Simples.
  <?php //################################ //# Magento Layered Nav - Fix Deux //################################ //# @author: Chris McKee [chrismckee.co.uk] //# @licence: GNU, basically do what you like but //# leave my name somewhere //################################ ob_implicit_flush(true); //Saves having to flush manually set_time_limit(0); //Set time limit to unlimited, though we shouldnt need to ignore_user_abort(); require_once("app/Mage.php"); umask(0); $total = 750; //How many records do we have (I'll make this automatic next time) $hm = 40; // How many records to process at once? $countFile = "cnt"; $store = null; $count = null; $store = Mage::app()->getStore('1'); //Create a count file if we dont have one if((file_exists($countFile))&&(filesize($countFile)>1)){ $fh = fopen($countFile, 'r'); $count = fread($fh,filesize($countFile)) + 1; fclose($fh); } else { $fh = fopen($countFile, 'w') or die("can't open file"); $count = 0; fwrite($fh, $count); fclose($fh); } //If we havent started counting yet clear cache. if($count == 0) { Mage::app()->cleanCache(); //Clear the database flag from any previous "normal" indexes that failed $flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf(); if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) { $flag->delete(); } } $max = $count+$hm; for($i = $count; $i <= $max; $i++) { Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $store); echo("\n #$i#CURRENT MEMRY USAGE = ".memory_get_usage().";\n"); //Write where were up to $fh = fopen($countFile, 'w') or die("can't open file"); fwrite($fh, $i); fclose($fh); sleep(1); //Pause a second, for less use u_sleep(microseconds) } ?>
  It takes about 1minute 2seconds for 40 records to process on my magento site using a total of 17mb of Memory. I’d generally advise running 10 and then calculating how much ram you are willing to sacrifice to the process minus 5%.
  Ok now to break down the code…
  Below we see a simple check, once we get past testing for the existence of our counter file and turning that into something useful we have a quick check. If its the first time this file has run we clean the cache. The second bit checks for the flag that Magento Cache management adds if its allowed to automatically run its Layered Navigation Cache (which fails, or you wouldnt be here). If the flag exists we kill it the same way magento would.
  if($count == 0) { Mage::app()->cleanCache(); //Clear the database flag from any previous "normal" indexes that failed $flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf(); if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) { $flag->delete(); } }
  This allows us to continue where we left off without wiping our previous work and allows us to fix anything we didnt want… but only where needed. Rather then wasting memory.
  Then for each loop of the process we add one to the counter file, we could change this to do it once at the end of the process loop but doing so would leave us without a counter in the event of a failure
  Simply run it from shell (or a script) as follows
  time php -q -f cm_LCache.php
  Time will output the time it took to run the script. -f is telling php to load the file. -q is quiet mode, but thats on by default.
  Let me know if anyone manages to make use of this.
  
  Trouble Shoot (BANG)
  A few problems I had due to the shared host I’m working on…
  - I created an empty text file in the same area as this script and set it to CHMOD 744 filename
  - Theres no method in here to reset the text file on completion but a quick and easy way is to simply have the script check for the max value being reached and use the updater from the loop but write a blank value and exit the script.
  - You dont need ssh to run this script as it should run as a backroung process, but there is no functionality to show you the progress builtin and most browsers will stop watching the page, so its something to consider.
  Download The PHP File

82 Responses to “ Magento Commerce – Layered Navigation Cache Error – Part Deux ”

Nicholas , on March 19th, 2009 at 5:38 pm Said:

Hey Chris,

What version are you running this script on? I have a 1.1.8 install I tried this on and am getting the following error kicked back:

Fatal error: Uncaught exception 'Exception' with message 'Notice: Use of undefined constant ÒdefaultÓ - assumed 'ÒdefaultÓ' in
/home/ridethis/public_html/app/code/core/Mage/Core/Model/App.php on line 575' in /home/ridethis/public_html/app/code/core/Mage/Core/functions.php:246 Stack trace: #0 
/home/ridethis/public_html/app/code/core/Mage/Core/Model/App.php(575): mageCoreErrorHandler(8, 'Use of undefine...', 
'/home/ridethis/...', 575, Array) #1 /home/ridethis/public_html/app/Mage.php(212): Mage_Core_Model_App-&gt;getStore(NULL) #2 
/home/ridethis/public_html/app/Mage.php(217): Mage::getStoreConfig('web/url/use_sto...', NULL) #3 
/home/ridethis/public_html/app/code/core/Mage/Core/Controller/Request/Http.php(118): 
Mage::getStoreConfigFlag('web/url/use_sto...') #4 /home/ridethis/public_html/app/code/core/Mage/Core/Model/App.php(270): 
Mage_Core_Controller_Request_Http-&gt;setPathInfo() #5 /home/ridethis/public_html/app/Mage.php(424): Mage_Core_Model_App-
&gt;init('', 'store', Array) #6 /home/ridethis/public_html/cm_cache.php(16): Mage::ap in 
/home/ridethis/public_html/app/code/core/Mage/Core/functions.php on line 246

Do you have any ideas about what might be going wrong?

Thanks for all the great work!

Chris McKee , on March 19th, 2009 at 11:37 pm Said:
Hi Nicholas,
Bizzare error there, where's it getting the constant default from...

The only thing that springs to mind is that you're not passing a necessary value at...
```
Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $store);
```
$i should be the product id (you can set this to null to process them all)
$store should be coming from $store = Mage::app()->getStore('1'); using your store id where 1 is.

Oh and I'm on 1.2.1 (the latest two upgrades after this not having any updates I'm interested in), with software as new as magento I'd advise trying to stay up to date with it, regardless of how irritating it is. Installing from svn helps. http://www.magentocommerce.com/svn

Chris
charlie , on April 12th, 2009 at 5:34 pm Said:

Chris,

should this theoretically resolve the issue where an attribute in the layered nav. is not showing the number of associated products (per this thread on magento http://www.magentocommerce.com/boards/viewthread/38685/)?

thx
Charlie
Chris , on April 16th, 2009 at 9:38 pm Said:

I used this script, and it worked great for me on a shared host and latest version of Magento and over 6000 Products that I imported. One thing I changed though was on the line:

sleep(1); //Pause a second, for less use u_sleep(microseconds)

I changed it to:

sleep(0); //No Pause

Works Great! Thanks for the great Script!

http://www.christmaseverymonth.co
Chris McKee , on April 16th, 2009 at 10:25 pm Said:

Hi, yes this fixes the problem in the thread. Basically if your layered cache doesn't finished you will either get zeros / an inaccurate count or a complete failure of the left hand nav.
Andreas , on April 24th, 2009 at 9:24 am Said:

Hi!

I have tried this script. And the script work without any problem. I have run it for over 5000 IDs now, but still won't the Layered Navigation work for me, can you tell me why? Why will it show up?

I run the script in a webbrowser because I don't have acess to my webhotel with ssh.

Should I make some manage in my cache properties or?

Great script, but I most get it to work for me!

//Andreas Andersson

Chris Nudd , on April 25th, 2009 at 6:35 pm Said:

Thanks for the script and email chris. Still getting this: Any ideas?

xxxxx@posdirect.com [~/public_html/eee]# time php -q -f cm_LCache.php
Failed loading /usr/local/Zend/lib/ZendExtensionManager_TS.so:  /usr/local/Zend/lib/ZendExtensionManager_TS.so: cannot open shared object file: No such file or directory
 
<b>Fatal error</b>:  Uncaught exception 'Zend_Db_Adapter_Exception' with message 'pdo_mysql extension is not installed' in /home1/posdirec/public_html/eee/lib/Varien/Db/Adapter/Pdo/Mysql.php:178
Stack trace:
#0 /home1/posdirec/public_html/eee/lib/Zend/Db/Adapter/Abstract.php(441): Varien_Db_Adapter_Pdo_Mysql-&gt;_connect()
#1 /home1/posdirec/public_html/eee/lib/Zend/Db/Adapter/Pdo/Abstract.php(230): Zend_Db_Adapter_Abstract-&gt;query('SET NAMES utf8', Array)
#2 /home1/posdirec/public_html/eee/lib/Varien/Db/Adapter/Pdo/Mysql.php(259): Zend_Db_Adapter_Pdo_Abstract-&gt;query('SET NAMES utf8', Array)
#3 /home1/posdirec/public_html/eee/app/code/core/Mage/Core/Model/Resource/Type/Db/Pdo/Mysql.php(45): Varien_Db_Adapter_Pdo_Mysql-&gt;query('SET NAMES utf8')
#4 /home1/posdirec/public_html/eee/app/code/core/Mage/Core/Model/Resource.php(85): Mage_Core_Model_Resource_Type_Db_Pdo_Mysql-&gt;getConnection(Object(Mage_Core_Model_Config_Element))
#5 /home1/posdirec/public_html/eee/app/code/core/Mage/Core/Model/Mysql4/Abstract.php(248): Mage_Core_Model_Resource in <b>/home1/posdirec/public_html/eee/lib/Varien/Db/Adapter/Pdo/Mysql.php</b> on line <b>178</b>
 
real	0m3.640s
user	0m0.106s
sys	0m0.034s

Chris McKee , on April 25th, 2009 at 8:17 pm Said:

@Andreas - You might want to take a look at the first post; You need to turn off Layered Cache in your admin (setting -> cache management -> untick layered cache) this stops it auto running and wiping your work.

@Chris Nudd - It looks like there's an error/issue in your magento install as I cant reproduce this error in any of my VMs

I'd look at the fact your first error is related to the zend optimizer lib not loading properly...
Failed loading /usr/local/Zend/lib/ZendExtensionManager_TS.so
Search

Seeing you have ssh access it might be worth checking the include exists )
Radek , on April 25th, 2009 at 8:33 pm Said:

Hello, can you specify where to put the script or where to create the file with this script? Well, I am a newbie... Thanx, Radek
Chris Nudd , on April 25th, 2009 at 8:45 pm Said:

I'm using bluehost. What's odd is that my phpinfo shows pdo loaded. http://www.posdirect.com/phpinfo.php.

Will continue to search the ZendExtManager.so... Thanks!
Chris McKee , on April 25th, 2009 at 11:31 pm Said:

@Chris Nudd: going off general searches for the errors blue host forums seems to be quite prominent.
@Radek: Place the file in the root of magento (basically the same folder as the root and .htaccess) it might help you to create a file called cnt on the server CHMOD 777 (write all)
Andreas , on April 26th, 2009 at 7:42 pm Said:

Thanks for the advice, but it still won't work... I have inactivate the layered navigations cache.

I am running Magento 1.3.0.

I can give you acess to my shop if you will check it and can found a solutions.

I am reall pleasured with all your help, Thanks!

//Andreas
len , on May 3rd, 2009 at 11:07 pm Said:

Thanks for this. Worked like a charm on version 1.3.1 for a little over 2000 products on the retail portion of my site at mydomain.com

However, on wholesale.mydomain.com (which is a second website using the same database) the layered nav is still broken. Any suggestions how to run the script for that website as well?

I assume it would involve altering $store = Mage::app()->getStore('1'); somehow but I'm not familiar w/ php at all so I'm pretty scared to mess around w/ it.
Chris McKee , on May 4th, 2009 at 5:09 pm Said:

@len: your right; you need to set the store id on that precise line to what ever your store ID is. You can get the ID from MagnetoStore admin -> System -> Manage Store -> click on the store and look at the URL (system_store/editGroup/group_id/1/) The bold bit is the store number )
Rod_C , on May 4th, 2009 at 10:44 pm Said:

I am also having problems with ZendExtensionManager_TS.so I am on a hostmonster server which is a sister company to bluehost.

I have checked the php.ini and it is enabled but till no luck. The responce I got back from there tec support was that its a problem with the script and not the servers. But due to the number of people with the exact same problem I doubt that. I will try contacting them again and if a come up with anything new I will post it back here.
Rod_C , on May 5th, 2009 at 11:29 pm Said:

Here's the reply from Hostmonster Tech Support:

Dear Customer, Thanks for contacting us.
Unfortunately you cannot use zend from the command line due to the way our systems work. You would need to execute these php files from a browser for it to use zend.
Thanks Mike Level 2 Scripting Support Engineer

Is it possible to run this script through a web browser ??
Chris Nudd , on May 6th, 2009 at 6:09 pm Said:

Had nightmare trying to run this on a shared host. Went to dedicated host and reindexed 8, 000 records in 46 seconds. Magento is just too ginourmous for shared hosts... Thanks Chris for everything!!!
Chris McKee , on May 7th, 2009 at 8:49 pm Said:

@Rod_C: Ah, the joys of web hosts. You CAN run this from the browser; in fact a lot of the emails I've had have been from people doing just that so feel free.

@Chris Nudd: Yep it is a tad greedy; but I can honestly say United Hosting uk - United Hosting US

I have a few clients there all on the second/third shared hosting package which comes with CHROOTED.

Los , on May 12th, 2009 at 6:40 pm Said:

I received the following error on 1.2.1 this error:

Fatal error: Uncaught exception 'Mage_Core_Model_Store_Exception' in /home/wwwadmin/public_html/DOMAIN/public/app/code/core/Mage/Core/Model/App.php:1194 Stack trace: #0 /home/wwwadmin/public_html/DOMAIN/public/app/code/core/Mage/Core/Model/App.php(608): Mage_Core_Model_App-&gt;throwStoreException() #1 /home/wwwadmin/public_html/DOMAIN/public/cm_cache.php(23): Mage_Core_Model_App-&gt;getStore('2') #2 {main} thrown in /home/wwwadmin/public_html/DOMAIN/public/app/code/core/Mage/Core/Model/App.php on line 1194

I also tried getStore('1') for the primary store and get the following error

Fatal error: Uncaught exception 'Zend_Db_Statement_Exception' with message 'SQLSTATE[42S22]: Column not found: 1054 Unknown column 'store_id' in 'field list'' in /home/wwwadmin/public_html/DOMAIN/public/lib/Zend/Db/Statement/Pdo.php:238 Stack trace: #0 /home/wwwadmin/public_html/DOMAIN/public/lib/Zend/Db/Statement.php(283): Zend_Db_Statement_Pdo-&gt;_execute(Array) #1 /home/wwwadmin/public_html/DOMAIN/public/lib/Zend/Db/Adapter/Abstract.php(457): Zend_Db_Statement-&gt;execute(Array) #2 /home/wwwadmin/public_html/DOMAIN/public/lib/Zend/Db/Adapter/Pdo/Abstract.php(230): Zend_Db_Adapter_Abstract-&gt;query('REPLACE INTO ca...', Array) #3 /home/wwwadmin/public_html/DOMAIN/public/lib/Varien/Db/Adapter/Pdo/Mysql.php(258): Zend_Db_Adapter_Pdo_Abstract-&gt;query('REPLACE INTO ca...', Array) #4 /home/wwwadmin/public_html/DOMAIN/public/app/code/core/Mage/CatalogIndex/Model/Mysql4/Indexer.php(363): Varien_Db_Adapter_Pdo_Mysql-&gt;query('REPLACE INTO ca...') #5 /home/wwwadmin/public_h in /home/wwwadmin/public_html/DOMAIN/public/lib/Zend/Db/Statement/Pdo.php on line 238

Chris McKee , on May 12th, 2009 at 10:14 pm Said:

@Los feel free to drop me a mail with more details (host : hosting type : OS : DB-TYPE) and anything else you think might be useful.

Ta
Sayed Haider , on June 1st, 2009 at 3:55 pm Said:

Hi Chris

I have the same problem with layered navigation. However, please pardon me as I am not very well versed with programing. Will you please be able to help me fix this for a little charge.

You can fix this once and then give me step by step instructions so I can run it in the future if needed.

I will really appreciate any help you can provide.

Thanks and best regards
Sayed
Ryan , on June 1st, 2009 at 5:19 pm Said:

Just in case anyone else is using mysql replication with different auto increment increments, this script assumes that ids go from 1 to $total. Our replication makes the IDs increment by 10 so I had to update the script to go by 10's instead of by 1. A better way would be to grab the actual ids from the database and then use those. Perhaps I'll do that later...
Chris McKee , on June 1st, 2009 at 8:54 pm Said:

@Ryan: The scripts based on the "General Standard" install of Magento; Obviously I based the count on the site I was working on at the time where I noticed that the records were incrementing in single units as per default behaviour. I did say/and I do need to finish writing this but as with most things in life paid work always comes first... it keeps me in tea.
Ryan , on June 2nd, 2009 at 7:22 pm Said:

@Chris yeah, I just posted that for the benefit of anyone else who might need it. Also, glad to see you'll be updating it soon! Thanks for the script! It saved us hours and hours and hours.
Carol , on June 5th, 2009 at 4:53 pm Said:

Worked like a charm!

Thank you!

Carol

James Allwood , on June 5th, 2009 at 8:46 pm Said:

Sorry, my mistake, corrected now, although now i get this error -



Fatal error: Uncaught exception 'Exception' with message 'Warning: fopen(cnt) [<a href='function.fopen' rel="nofollow">function.fopen</a>]: failed to open stream: Permission denied in /home/petshome/public_html/canistercountry/magento/layeredupdate.php on line 33' in /home/petshome/public_html/canistercountry/magento/app/code/core/Mage/Core/functions.php:239 Stack trace: #0 /home/petshome/public_html/canistercountry/magento/layeredupdate.php(33): mageCoreErrorHandler(2, 'fopen(cnt) [&lt;a ...', '/home/petshome/...', 33, Array) #1 {main} thrown in /home/petshome/public_html/canistercountry/magento/app/code/core/Mage/Core/functions.php on line 239

Any help would be appreciated. Thanks

James Allwood , on June 5th, 2009 at 8:28 pm Said:
Hi. Well I think this is what I've been looking for. I've imported 15,000 products and all went well, but when trying to refresh layered navigation I just end up with a blank screen. Anyway after running this script I get this strange error-
```
Parse error: syntax error, unexpected T_VARIABLE in /home/petshome/public_html/canistercountry/magento/layeredupdate.php on line 4
```
Anyone any ideas? Thanks
Chris McKee , on June 6th, 2009 at 10:25 am Said:

@James Allwood the error is very much in the error message; file permissions aren't set correctly for file cnt; either set the file cnt to CHMOD 0777 or if it doesnt exist create it and set it to 777. I unfortunately could only base the instructions on my own experience.
James Allwood , on June 6th, 2009 at 11:14 am Said:

Hi Chris, thanks so much for the help. In the frustration of everything, it looks like I hadn't understood everything correctly.

Anyway, it runs now, but I seem to get the same result as refreshing from admin, after a few minutes I just end up with a blank screen. I've looked in cnt and it displays '0'. Any further ideas? I'm desperate to use Magento, but this is plain and simple a dead end. Thanks
Chris McKee , on June 6th, 2009 at 2:21 pm Said:

@James Allwood: make sure that layeredupdate.php is set to 755; and if your trying to run it from the browser you will need to keep the numbers low as the browser will time out.
Ideally this would be run from a shell command not from a browser; I'm still a little confused why so many seem to have opted for small-site hosting packages for such a large application... Anyhoo, as with all, let me know how you get on.
James Allwood , on June 6th, 2009 at 7:57 pm Said:

Hi again Chris. Well massive thanks for your help again, I did have the wrong file permissions, but now that they're changed I still get a blank screen after a couple of minutes. I'm trying to set up SSH to have a crack at it that way. I'm actually using a half decent hosting package from siteground, which should really have plenty of resources to do the job, but I may be proved wrong? Anyway, I'll let you know how I get on with SSH, and if anyone should have any other suggestions I'd be very grateful. Thanks
Chris McKee , on June 7th, 2009 at 11:03 am Said:

@James Allwood: I'd say have a pop at running it in SSH; make sure error output is on; or set it manually in the file.
I did tracert your domain to see who the host was and yep it seems pretty decent; VPS/DPS you should be fine.
James Allwood , on June 7th, 2009 at 11:13 am Said:

@Chris Kckee: Just got SSH set up this morning, and used the command you've provided to run it, but it seems to just hang. I left it running for a while, and then checked cnt, which still contained '0'.

The command I used was - time php -q -f layeredupdate.php

I press enter and nothing happens? Thanks Chris
James Allwood , on June 7th, 2009 at 11:48 am Said:

@Chris Kckee: Hold on to your socks, as it looks like the script has just randomly started working. It ran through 40 records and then stopped. Just going to have a crack at changing those values and running again. Will report back, with hopefully 'success'!
Chris McKee , on June 7th, 2009 at 12:20 pm Said:

@James Allwood: You'll notice the cnt file keeps a tally of where you are; you just run it multiple times.
Basically the layered cache leaks memory like a sieve so the only way to keep it safe was to run a certain amount then terminate and run where you left off.
Stook , on June 10th, 2009 at 5:12 am Said:

This worked great for a while and then i had the same problem as Chris Kckee. But mine's still not working. it just leaves a blank curser.
Chris McKee , on June 10th, 2009 at 11:18 am Said:

@Stook Same problem as me? I take it you mean a post I've replied to, let me know which.
As for killing the process; it'll either timeout in the browser or you can press CTRL+C in SSH to kickout of the process; or you can kill the process manually with kill http://speculation.org/garrick/kill-9.html
Guy de Nooij , on June 13th, 2009 at 7:03 pm Said:

Thanks Chris for this script which seems to help a lot of people. I am getting an error, well I'm not even sure that it is an error. I run the script in my browser and after the first 50 records this text appears: #51#CURRENT MEMRY USAGE = 9821000; and continues with #52 and so on.
I think my memory runs out but I don't see any errors in the serverlog. Any ideas or suggestions would be appreciated.
Chris McKee , on June 14th, 2009 at 5:21 pm Said:

@Guy de Nooij: Reduce the number your processing; in command line the program outputs the number its on and the memory usage every line. in the browser, buffering stops this so only the last line is displayed.
Jack Rose , on June 19th, 2009 at 12:46 pm Said:

Thanks for this - worked a treat. Is there something similar I could use to rebuild the search index as I get "Fatal error: Maximum execution time of 30 seconds exceeded" whenever I use the rebuild button in admin?
E , on June 20th, 2009 at 9:32 am Said:

I would be interested in something like a search index rebuild script too. And also for catalogue rewrites (what does that even do?) and catalogue index (same question).

Or a complete rebuild of Magento ... pff
Chris McKee , on June 22nd, 2009 at 11:37 am Said:

@E: Ha; not sure I could stretch out to a complete rebuild of Magento. Catalogue rewrites cache is the lookup table for URL Rewriting.
Tanya Kumar , on June 25th, 2009 at 12:23 pm Said:

Chris, I have followed the instructions, unchecked Layered Navigation in the cache managed, made the file, uploaded to root, gave 777 permission. However, when I try to run the file from the 'browser' i get a 500 Internal error. Please help!
Tanya Kumar , on June 25th, 2009 at 12:54 pm Said:

The script creates a file on the server called cnt - this is a 1KB file with the value 0 in it - I think I'm missing something
Chris McKee , on June 29th, 2009 at 5:24 pm Said:

@Tanya Kumar; Make sure the script is in the correct place (i.e file.php is in the same folder as the app folder and .htaccess); Set the php file to CHMOD 0755.
The count file is there to keep a running total so that you can update the indexes in segments rather than all at once.
anastasia - Greece , on June 30th, 2009 at 9:16 am Said:

Chris you are GREAT!!! THANK YOU for the script!!!!
Works perfectly!!!
Chris McKee , on June 30th, 2009 at 8:05 pm Said:

Thanks; Glad to hear it worked.
If people could stick their version numbers (Magento) in the message it would be appreciated as its handy to know what its been tested on (in functional sites; i only test it in clean installs)
John Gills , on June 30th, 2009 at 8:13 pm Said:

Is anyone else experiencing a problem with the script timing out / stopping in the loop at this step?

Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $store);

I'm on hostmonster, (bluehost).
I'm running the script through the browser because of the ZendExtensionManager_TS.so issue mentioned above.
Magento 1.2.0.1
Chris McKee , on July 1st, 2009 at 8:10 pm Said:

@John Gills: quite a few emails I've had relating to problems with magento + running this script have been people hosted with bluehost; Unfortunatly All I can offer in the way of advise is to move to a host that specifically says its suited for magento.

http://www.unitedhosting.co.uk/uk-magento-web-hosting.php
http://www.magentocommerce.com/partners/find/hosting-partners/
Jake Buhrman , on July 5th, 2009 at 4:53 am Said:

Chris, Works perfect on SiteGround Shared host processing 500 records at a time. 2,000 products (so far) running Magento 1.3.2.1. Can you tell me what table I can look at in the database that will give me some indication that all the records have been indexed?
JK , on July 5th, 2009 at 7:41 pm Said:

Hi, my friend asked my to repair his shop because he got problems with his layered navigation in Magento store.
I was searching over the internet and I found one FIX created by Chris McKee. THX Chris
His script was perfect but some automatic functions were missing, therefore I’ve created my own script.
Check it here: http://jksite.com/blog/2009/07/magento-layered-navigation-cache-error
Many THX Chris!
Tanya Kumar , on July 7th, 2009 at 1:14 pm Said:

Still no luck Chris the location is correct. Permissions are correct. FYI I am hosting it on a shared 1and1 linux hosting... could that be the problem?
Chris McKee , on July 7th, 2009 at 3:12 pm Said:

@Tanya ; Might be worth asking 1&1, though they are notoriously recognized for their interesting concept of customer service, they're at least likley to know their own setup.

I'd always advise either using a container type host (united hosting style) or a vps (or if your loaded a DS)
John Gills , on July 9th, 2009 at 4:45 pm Said:
@Chris - I've been successful with a modification to your script. I'm not sure what the difference is that is making it work on HostMonster but in case someone else is running into the same issue, i wanted to provide it here:
http://www.magentocommerce.com/boards/viewthread/11938/P60/#t120108

That said, I have an additional question. What records are you interating through to establish the index on line 52?
i.e.

<?php for($i = $count; $i plainReindex($i, null, $store); echo("\n #$i#CURRENT MEMRY USAGE = ".memory_get_usage().";\n"); //Write where were up to $fh = fopen($countFile, 'w') or die("can't open file"); fwrite($fh, $i); fclose($fh); sleep(1); //Pause a second, for less use u_sleep(microseconds) } ?>

And thanks for your help. You saved me a lot of frustration.
Chris McKee , on July 9th, 2009 at 10:07 pm Said:
Line 52 being :
for($i = $count; $i <= $max; $i++)
$count is the contents of the cnt file (kind of a marker of our position so far) $i is just the current position (starting at $count) in the loop. $max is $count + $hm; so our current position + $hm (records to process in one go) The loop runs over the Single Item Reindex singleton
Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $store);

Which is usually used when you update a product to update the layered-nav without rerunning the index. So this is a quick, lazy and easy way to abuse that method. The script needs to run in small chunks due to the memory leaks that occure in Mage.

I'm currently working through the index class to pick it apart as I'm sure I could write this more efficiently without all this faff. Ideally as a database script rather then buggering about in PHP.
E. , on July 12th, 2009 at 5:35 pm Said:

Chris, I've got a question.

I just did a testrun of script on my 20.000 products store, and (I think) it took 155 minutes (real 155m43.328s user 93m54.400s sys 2m11.068.s).

I've put the sleep to 0, and I noticed that for most of the time the terminal just 'freezes' (meaning you cant see any process / changes) and then runs trough the records really fast.

Is the script while frozen actually removing the old cache (what could be taking a long time) and then rebuilding the cache in a couple of minutes? What does it do when the count file isn't yet there or set to 0?

My configuration is as following (always worked perfectly):
$total = 0
$hm = 20000
sleep(0);

alamgir , on July 13th, 2009 at 5:28 pm Said:

Layered navigation from the backend is unchecked. I chmod the file to 0755 and put it on the root directory of magento. When I run this script from the browser I get this error.

Fatal error: Uncaught exception 'Mage_Core_Model_Store_Exception' in /home/pcrepai2/public_html/mydomain.com/app/code/core/Mage/Core/Model/App.php:1267 Stack trace: #0 /home/pcrepai2/public_html/mydomain.com/app/code/core/Mage/Core/Model/App.php(633): Mage_Core_Model_App-&gt;throwStoreException() #1 /home/pcrepai2/public_html/mydomain.com/cm_cachephp.php(16): Mage_Core_Model_App-&gt;getStore('1') #2 {main} thrown in /home/pcrepai2/public_html/mydomain.com/app/code/core/Mage/Core/Model/App.php on line 1267

I am using Magento (ver. 1.3.2.2)....Any help would be great..Thank you.

Chris McKee , on July 13th, 2009 at 9:18 pm Said:

@alamgir: make sure your using the text file to copy paste to avoid any obvious issues (like me cocking up editing the post yesterday and wiping half the code)

@E: due to the way PHP buffers responses, chances are its processing a pile of stuff first then outputting the actual messages later (making it look a lot faster than it is)
alamgir , on July 13th, 2009 at 10:19 pm Said:

It's working now. so far I refreshed about 1700/4000... The problem was the store ID, so I changed it and now so far it's working perfectly.

Also, I am able to do $hm = 240; at once without any problem.

Thank you so much for this script. You just saved me hours of work. Again thank you so much...
Chris McKee , on July 13th, 2009 at 11:49 pm Said:

@alamgir: Glad to hear it.
Slava , on July 15th, 2009 at 5:12 pm Said:

Hi, i try use script and get 500error after about 10-20sec. I have 4488 product, i copy you script, make new php file, change $total = 4488 and leave $hm = 40. I don't have SSH. I try run from browsers and try run with cron job and get same 500error, check permission try 755 and 777, nothing help. In error log (Plesk) see:
[error] Premature end of script headers: cm_cache.php
My server use FastCGI php. Her phpinfo http://www.apolon.lt/phpinfo.php Can you help me find the problem, thanks!
Chris McKee , on July 16th, 2009 at 1:10 pm Said:

@Slava: you dont say if your running it from the browser or SSH, so I'll assume browser. Did you copy the script from the text file (http://chrismckee.co.uk/crapdump/2009/03/cm_cachephp.txt) as some text-editors/web-editors can misinterpret copy-paste webpage text.
Failing that make sure php errors are turned on and send back the error that occurs.

Thanks
Slava , on July 17th, 2009 at 5:18 pm Said:

i run from browser. I upload you textfile and renamed to php. I try turn on all error in script
ini_set('display_errors',1);
error_reporting(E_ALL|E_STRICT);
but nothing same error 500Intenal Server Error
Slava , on July 21st, 2009 at 1:25 pm Said:

I get from hoster mail, like this script use to much server resource and need update him separate jobs
Chris McKee , on July 21st, 2009 at 1:57 pm Said:

@slava; reduce the number your processing at a time. I see your on shared hosting so you may want to consider another hosting platform if they wont supply enough processor time to run a recursive sql query.
Slava , on July 21st, 2009 at 4:19 pm Said:

After this script price sort work? any body can check? Thanks
Mark H , on July 22nd, 2009 at 3:44 pm Said:

I don't understand what i'm doing wrong... I have followed everything here, and i'm running this in a browser, I don't get an error but a screen comes up saying many lines lik

#4373#CURRENT MEMRY USAGE = 4217200;

I only have 2400 products so i don't understand why it passed this number, and my layered index still have all zeros...

Am i supposed to sit here and keep refreshing the browser? I don't understand what to do, please help...
Chris McKee , on July 22nd, 2009 at 3:58 pm Said:

@Mark H: Make sure you've set the max to the actual number of records, and make sure you've got the right store code in ($store = Mage::app()->getStore('1');)
Mark H , on July 22nd, 2009 at 6:54 pm Said:

the actual amount is something like 23XX, should it be absolutely exact? either way its going way past 2400... into the 4000's... Whats also weird is i used the code from the magento forums, and then i tried the code from this page, both set to 2400 records and store set to 1, and when i tried using the second, it just continued where the first left off, i guess the code is pretty similar...

anyway, what should i be seeing? how should i know when its finished? am i supposed to have to keep refreshing?
Chris McKee , on July 22nd, 2009 at 8:30 pm Said:

@Mark the code on the magento forum will be the code off this site so not so much similar as the same.

If you want to start from scratch just delete or empty the cnt file.

Ideally you should run this script over SSH in BASH/Shell (etc); running from browser, mileage may vary.
Will , on July 23rd, 2009 at 2:55 am Said:

Chris, you magnificent bastard, you saved my arse!

At first it didn't work because even though I have 393 products, my Product IDs don't go from 1 to 393. I guess this is because I deleted a bunch of stuff. Anyway they start at ID 82. To get your script to work I had to change the number in cnt from 0 to 81. And the cm_cache total to 477.

Future players also note that if you do it this way you may want to manually delete any core_flag fields in your database, cos I think Chris's script only does that if the cnt file is 0 (Chris, can you confirm/deny this?).

Thanks again!
Will.
Mark H , on July 23rd, 2009 at 6:51 am Said:

The thread i was looking at was a little different, perhaps the person changed it... however, i'd like to start over but i'm not sure where the file you're referring to..

and sorry, but i'm i've a bit of a newbie in this area... how do i run something in SSH?

Chris McKee , on July 23rd, 2009 at 9:28 am Said:

@will: yep the coreflag is just a check count hasn't started before clearing; you could easily change...



< ?php
if($count == 0)
{
    Mage::app()->cleanCache();
    //Clear the database flag from any previous "normal" indexes that failed
    $flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
    if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) {
        $flag->delete();
    }
}
 ?>



< ?php
//IF Count Zero or Querystring passed (thisfilename.php?clear=1)
if(($count == 0)||( $_GET['clear'] == "1"))
{
    Mage::app()->cleanCache();
    //Clear the database flag from any previous "normal" indexes that failed
    $flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
    if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) {
        $flag->delete();
    }
}
?>

The codes due to be updated I'm just on a go slow. This could also be passed as an Argument rather than a querystring for people using SSH. Have a look at the previous code (Part one) for details.

@Mark: Use the contents of the downloadable file in this article; upload; download a client such as PUTTY and login to your server with your user pass host address. You should then be able to use commands like "time php -q -f cm_LCache.php" If your on shared hosting this may not be available.
A lot of people have managed to run this fine from the browser (in fact it was kind of made for it.)
New version out VERY soon now as I'm pretty much done. I just need to test it.

Chris

Mark H , on July 23rd, 2009 at 6:29 pm Said:

Ok, cool i will download PUTTY... but what does this mean? I have no idea...

“time php -q -f cm_LCache.php”
Mark H , on July 23rd, 2009 at 10:09 pm Said:

i get no input file specified
Mark H , on July 23rd, 2009 at 10:22 pm Said:

i don't get what the cnt file is... do i create a txt file named cnt? what kind of file is this...
Mark H , on July 23rd, 2009 at 11:03 pm Said:

I found the cnt file and deleted it for a fresh start...

my magento installation is not in the root of the hosting folder... so my php file is in /magento/layerednavfix.php

what should i type for it to run in SSH, i tried time php -q -f layerednavfix.php and time php -q -f magento/layerednavfix.php and neither worked... both say no input file specified... what am i doing wrong...
Chris McKee , on July 24th, 2009 at 9:19 am Said:

Hi Mark;
Lot of questions in a row there ;o)

cnt is technically just a text file; as this is linux it doesn't really matter and not specifying a filetype is better practice as it means the file doesn't have a default mime.

SSH in to the site, CD magento (change directory to magento)
and try just typing php --help and see if you get the help file.
On some its php-cli on others its php-cgi; 90% of the time its just php.

The options are as follows: http://www.php-cli.com/php-cli-options.shtml

time = time this process
php = program
-q =
-f = Parse .
-q is quiet mode
Chris McKee , on July 24th, 2009 at 10:59 am Said:

New Release ...
http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-badger-part-duck/

As this post may still be of use to certain people I'll leave comments open.
andrew , on November 12th, 2009 at 2:15 am Said:

I have used the browser version and it seems to be successful. I am trying to use this to be able to sort my products by price once again. Even after running this, the products won't sort correctly. Do you think I am doing something wrong with this script or is it possibly another error?
andrew , on November 12th, 2009 at 2:21 am Said:

sorry, meant to put that on the third version. will post there now.
Fernando , on January 1st, 2010 at 11:58 pm Said:

Thankyou so much Chris , saved my life!!!! and Happy new Year!!!

» Magento Commerce – Layered Navigation Cache Error – Part Deux | Chris Mckee Designs, by Chris McKee, who'd of guessed | ChrisMcKee.co.uk

Magento Commerce – Layered Navigation Cache Error – Part Deux

Solution 2 – Running the Layered Navigation Cache in bits from PHP-CLI

Trouble Shoot (BANG)

82 Responses to “ Magento Commerce – Layered Navigation Cache Error – Part Deux ”

Leave a Reply