» Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck | Chris Mckee Designs, by Chris McKee, who'd of guessed

Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck
July 24, 2009

Solution 3 – Running the Layered Navigation Cache; PHP-CLI & Browser Version [fixed - Again]

Magento 1.4 will probably solve this issue for many people, so it may be worth giving it a squirt )

Fuck up central, exporting the script caused some of the comments to drop to multiple lines. If you had problems re-download please.
Browser edition now updated to actually delete files on completion and in clicking complete deleting the files and stopping the refreshes.
———————————-
Ok so I buggered up the first release (hence the post dissapearing), Basically I’d “counted” the products rather than grabbing the last ProductID as used by the Singleton method. So I’ve resolved that (thanks Mark H for pointing it out, I must of been half asleep). Processing through none existant records has no detrimental effect (it checks and jumps them) so I’ll skip on the suggestion of running through the array, thanks.

Well here goes, part three… Deep Breath.

First some specs:

On my test server (restrictions as on shared hosting in CHROOT);
100 records took this long

Previous 100 records in

#Record 101#CURRENT MEMRY USAGE = 152,411,480;
real    5m37.504s
user    1m19.869s
sys     0m3.362s

New 100 records in

#Record 101#CURRENT MEMRY USAGE = 40,411,480;
real    2m30.101s
user    0m34.841s
sys     0m0.907s

New Features

*NEW* PHP-CLI Edition
* Auto Product Count
* Auto On Completion Destroy Temp Files
* Various Optimizations

*NEW* PHP Web Browser Edition
* Three files;
** Wrapper (which you run from your browser)
** Two IFRAMES: The main functional script and a counter to show its not fallen asleep
** Auto Updating output; set off and sleep mode p

Now for the breakdown (script not mental)

Magento Layered Cache Fix PHP-CLI Edition

#!/usr/bin/php -q
<?php
//#######################################
//# Magento Layered Nav - Badger || Duck
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# MLBv3.1.1
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
try{
	ob_implicit_flush(true); //Saves having to flush manually
	if(!ini_get('safe_mode')){set_time_limit(0);}/*Set time limit to unlimited, though we shouldnt need to*/
	ignore_user_abort(); //Lets the script run to completion even if you kill the browser
 
	require_once("app/Mage.php");
	Mage::app('admin'); //as per magento handling
	Mage::setIsDeveloperMode(true); //apmh
	umask(0);
 
	/*{ CONFIGURABLES }*/
	$HM = 800; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
 
	/* Automatically Count Products */
		$total = NULL;
		if($HOWTORUN == 1)
		{
			/*## CLI Version */
			$prodCount = "cm_pd";
			if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
			else
			{
					$fh = fopen($prodCount, 'w') or die("can't open file");
 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
 
					$total = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
 
					fwrite($fh, $total);
					fclose($fh);
			}
			/*## End CLI Version*/
		}
		else
		{
			/*## Browser Version */
			if((!isset($_SESSION['pcount']))||($_SESSION['pcount']==NULL)){ 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$endValue[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
					$_SESSION['pcount'] = $endValue;
				$total = $_SESSION['pcount'];
			}
			else{ $total = $_SESSION['pcount']; }
			/*## End Browser Version */
		}
		if($total == NULL){ throw new Exception("Total Wasn't Captured"); }
	/* End Count Products */
 
	echo("\n$total\n");
 
	$countFile = "cm_cnt";
	$count = null;
	$store = Mage::app()->getStore($STOREID);
 
	//Create a count file if we dont have one
	if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	else
	{
			$fh = fopen($countFile, 'w') or die("can't open file");
			$count = 0;
			if((int)$count == 0) $count = 1;
			fwrite($fh, $count);
			fclose($fh);
	}
 
	//If we havent started counting yet clear cache.
 
	if($count < 2)
	{
			Mage::app()->cleanCache();
			//Clear the database flag from any previous "normal" indexes that failed
			$flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
			if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) $flag->delete();
	}
 
	$max = ($count+$HM);
	$sx = Mage::app()->getStore('1');
 
	ob_start();
	echo "start loop (max = $max) (count = $count)";
 
	for($i = $count; $i <= $max; $i++)
	{
	ob_end_flush();
	ob_start();
				if($i == $total)
				{
						echo "<br><h1>Completed</h1><br>";
						@unlink($prodCount);
						@unlink($countFile);
					exit(1);
				}
 
 
			Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $sx);
 
			echo("\n #Record $i#CURRENT MEMRY USAGE = ".memory_get_usage().";\n"); //UPDATE SCREEN
 
			//Write where were up to
			$fh = fopen($countFile, 'w') or die("can't open file");
			fwrite($fh, $i);
			fclose($fh);
	ob_flush();
	flush();
	}
 
}catch(Exception $e){ echo"<br><p style=red>ERROR: $e</p><br>"; exit(1);}
#EOF
?>

Mostly though you’ll need to change this bit to run it in PHP-CLI over SSH…

/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/

How many records to process at once: I comfortably managed 500 at a time without it breaking a sweat
StoreID: I’ll let you guess, but for most of us mortals 1 is the default.
HowToRun: This is extendable so basically the guts is there to allow you to use Sessions (in the browser version) IO readings slightly faster so in the end it won over sessions

Thats pretty much it, aha, yep thats all, stop asking.

Simply run it from shell (or a script) as follows

time php -q -f cm_LayeredCache_badger.php

Time will output the time it took to run the script; -f is telling php to load the file; -q is quiet mode, but thats on by default.

Magento Layered Cache Fix Browser Edition

badger-cm_LayeredCache.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
try{
	ob_implicit_flush(true); //Saves having to flush manually
	if(!ini_get('safe_mode')){set_time_limit(0);}/*Set time limit to unlimited, though we shouldnt need to*/
	ignore_user_abort(); //Lets the script run to completion even if you kill the browser
 
	require_once("app/Mage.php");
	Mage::app('admin'); //as per magento handling
	Mage::setIsDeveloperMode(true); //apmh
	umask(0);
 
	/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
 
	/* Automatically Count Products */
		$total = NULL;
		if($HOWTORUN == 1)
		{
			/*## CLI Version */
			$prodCount = "cm_pd";
			if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
			else
			{
					$fh = fopen($prodCount, 'w') or die("can't open file");
 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$total = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
 
					fwrite($fh, $total);
					fclose($fh);
			}
			/*## End CLI Version*/
		}
		else
		{
			/*## Browser Version */
			if((!isset($_SESSION['pcount']))||($_SESSION['pcount']==NULL)){ 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$_SESSION['pcount'] = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
				$total = $_SESSION['pcount'];
			}
			else{ $total = $_SESSION['pcount']; }
			/*## End Browser Version */
		}
		if($total == NULL){ throw new Exception("Total Wasn't Captured"); }
	/* End Count Products */
 
	echo("\n$total\n");
 
	$countFile = "cm_cnt";
	$count = null;
	$store = Mage::app()->getStore($STOREID);
 
	//Create a count file if we dont have one
	if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	else
	{
			$fh = fopen($countFile, 'w') or die("can't open file");
			$count = 0;
			if((int)$count == 0) $count = 1;
			fwrite($fh, $count);
			fclose($fh);
	}
 
	//If we havent started counting yet clear cache.
 
	if($count < 2)
	{
			Mage::app()->cleanCache();
			//Clear the database flag from any previous "normal" indexes that failed
			$flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
			if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) $flag->delete();
	}
 
	$max = ($count+$HM);
	$sx = Mage::app()->getStore('1');
 
	ob_start();
	echo "start loop (max = $max) (count = $count)";
 
	for($i = $count; $i <= $max; $i++)
	{
	ob_end_flush();
	ob_start();
				if($i == $total)
				{
					echo "<br><h1>Completed</h1><br>";
					exit(1);
				}
			Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $sx);
 
			//Write where were up to
			$fh = fopen($countFile, 'w') or die("can't open file");
			fwrite($fh, $i);
			fclose($fh);
	ob_flush();
	flush();
	}
 
}catch(Exception $e){ echo"<br><p style=red>ERROR: $e</p><br>"; exit(1);}
#EOF
?>

badger-count.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck - Counter
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
?>
<html>
<head>
<?php if(!isset($_GET["done"])){ ?>
<meta http-equiv="refresh" content="5" />
<?php } ?>
</head>
<body>
<?php
$prodCount = "cm_pd";
$countFile = "cm_cnt";
 
 
if(isset($_GET["done"])){ 
	unlink($prodCount); 
	unlink($countFile);
	echo "Files Deleted";
	exit(1);
}
 
if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	if(isset($total)){
		$left = ($total - $count);
		echo "<b>[Count: $count | Total: $total][ $left to go]</b>";
				if($left == 0)
				{
					echo "<br /><a href='badger-count.php?done=1' target='_parent'>Click Here To Complete and Delete Count files</a>";
					exit(1);
				}
	}
?>
</body>
</html>

badger-wrapper.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck - WRAPPER
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-gb">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Magento Layered Cache</title>
<meta http-equiv="robots" content="noindex,noarchive" />
<style type="text/css">
<!--
@charset "utf-8";
/************************************************************\
 * Core: Magento
 * @Target: Layered Nav refresh
 * @author: Chris.McKee
 * @date: 23-07-2009
\************************************************************/
 
html{ font-size:16px;color:#000; background:#F9F9F9; } /* Set BASE Font Size */
 
/* Reset Browser Differences */
*,body,div,dl,dt,dd,ul,ol,li,h1,h2,h3,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0px;}
table{border-collapse:collapse;border-spacing:0}
address,caption,cite,code,dfn,em,th,var{font-style:normal;font-weight:400}
ol,ul,li{list-style:none}
caption,th{text-align:left;}
h1,h2,h3,h4,h5,h6{font-size:100%;font-weight:400}
q:before,q:after{content:''}
fieldset,img,abbr,acronym{border:none;font-variant:normal;}
sup{vertical-align:text-top;}
sub{vertical-align:text-bottom;}
input,textarea,select{font-family:inherit;font-size:inherit;font-weight:inherit;}
input,textarea,select{*font-size:100%;}
legend{color:#000;}
html, body { height:100%;	width:100%;	}
/*EORESET*/
/* Now thats over we can begin */
body {font:0.8em "Trebuchet MS", Helvetica, Verdana, Arial, FreeSans, sans-serif;}
h1{
	font-size:1.6em;
	position:absolute;
	left:4px;
	display:block;
	top: 8px;
}	
#refresh{
  position:absolute; 
	left:-1000px;
	height:0px;
	width:0px;
}
#counter{ position:relative; margin: 25% auto 0px; border:0px; outline:0px;}
#counter * {font-size:5em; text-align:center; }
small a{ position:absolute; bottom:8px; right:8px; display:block; text-decoration:none; color:#666; }
 
-->
</style>
<script type="text/javascript">
//<![CDATA[
//40 Seconds (no point in changin this)
var intervalID;
function startREFRESH(){  intervalID = setInterval(reloadREFRESH, 40000); }
function reloadREFRESH(){ refresh.location.reload(1); }
function stopREFRESH(){ clearInterval(intervalID);}
//]]>
</script>
<link rel="Shortcut Icon" type="image/x-icon" href="http://chrismckee.co.uk/favicon.ico" />
</head>
 
<body onload="startREFRESH();">
<h1>Layered Navigation Refresh From Browser</h1>
<iframe src="badger-count.php" id="counter" name="counter">You At Least Need a Browser that Supports IFRAMES</iframe>
<iframe src="badger-cm_LayeredCache.php" id="refresh" name="refresh" >You At Least Need a Browser that Supports IFRAMES</iframe>
<small><a href="http://chrismckee.co.uk/?s=layered+navigation">ChrisMckee Designs</a></small>
</body>
</html>

/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/

How many records to process at once: Dont Bother changing this
StoreID: I’ll let you guess, but for most of us mortals 1 is the default.
HOWTORUN: As with the CLI version you can ignore this

To run simply upload to the root of the magento site (as with all the other instructions) and open the badger-wrapper in your browser )

Let me know how you get on and and issues

Previous Versions

DOWNLOADS (updated 20090728)

20090728: Line: if($count == 0) changed to if($count < 2) otherwise condition never met; meaning if your indexes were running (on the cache page your "KILL" option was still there) the script would run, but MAGE would not store your indexed results.
PHPCLI Layered Cache – vbadger – Zipped PHP File
Browser Layered Cache – vbadger – v1.3.4

TROUBLESHOOTING BANG

Many people on shared hosting experience the following error

Failed loading /usr/local/Zend/lib/ZendExtensionManager_TS.so: /usr/local/Zend/lib/ZendExtensionManager_TS.so: cannot open shared object file: No such file or directory

In this case it’s apparent that your CHRooted/shared environment isn’t set to allow direct access to Zend as required by Magento; in this case its often a good idea to contact your Web Host to see if this is something they can resolve for you.

To show its not this script that causes the issue try the following standard import of Mage…

<?php
//Create a file called testmage.php and add the following code and run. php -qf testmage.php via SSH
//Errors don't always appear in shared environments when using a browser.
require_once("app/Mage.php");
?>

The only “solution” is to run the browser version which acts more like Magento does, though with this mileage may vary

Thanks to Oleksiy for letting me into his server to diagnose this issue.

#2 The indexing finishes but my layered navigation doesn’t appear: I’ve heard this one 5 times now and happily its always the same problem… It’s not appeared because you’ve not turned it on.

Required attribute conditions to be shown on layered navigation filters:

Attribute properties / Catalog Input Type for Store Owner = Dropdown or Multiple Select (or the System Attribute, Price)
Frontend properties / Use In Layered Navigation = Filtered *
System properties / Data Type for Saving in Database = Integer
System properties / Globally Editable = Yes
Please note that a category should have the ‘Is Anchor’ property set to Yes in order for the layered navigation block to show.

Source: http://www.magentocommerce.com/wiki/general/layered-navigation

Share |

141 Responses to “ Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck ”

Mike D , on July 23rd, 2009 at 10:11 pm Said:

Hey Chris,
I was fixing to try this out. Looks like this will not get the correct product ids if not starting from 1 and incrementing by one.
I modified this code to grab the product ids as an array.
you can also count the collection size by using "getSize". for what its worth
getCollection()->getAllIds();Mage::getSingleton('catalogindex/indexer')->plainReindex($productIds[$i], null, $sx);
Problem resolved (CMCKEE - 24-07-09); Thanks for the feedback
JK , on July 24th, 2009 at 11:40 am Said:

Hi Chris, many THX for update!
If you will find any of the free hosting that can be used for Magento please let me know. Thanks
Chris McKee , on July 24th, 2009 at 6:19 pm Said:

Cool; let me know how you get on.

.
Chris McKee , on July 25th, 2009 at 9:33 pm Said:

Scripts Updated, PLEASE DEAR GOD TELL ME IF YOU HAVE PROBLEMS WITH IT...

Thanks )

ogonkov , on July 27th, 2009 at 2:06 pm Said:

Ssh ver. isn't working for me, i got:

$ time php -q -f cm_cache.php
PHP Warning:  Unknown(): Unable to load dynamic library '/usr/local/lib/php/20020429/mssql.so' - /usr/local/lib/php/20020429/mssql.so: Undefined symbol &quot;OnUpdateLong&quot; in Unknown on line 0
 
Parse error: syntax error, unexpected '{' in /www/avrorus/www/htdocs/products/cm_cache.php on line 12
 
real    0m0.220s
user    0m0.008s
sys     0m0.005s

ogonkov , on July 27th, 2009 at 2:37 pm Said:

I'll try now browser version, it's run very good, but i don't see counter of how much records done, just header and small link to you site at bottom. Only thing i change was howtorun 2 in badger-cm_LayeredCache.php, i shouldn't do it?
Chris McKee , on July 27th, 2009 at 2:41 pm Said:

No as the comments say dont change the how to run, the only confugurables are store id (as listed in the article)

Chris McKee , on July 27th, 2009 at 2:49 pm Said:

PHP CLI VERSION TEST -->
Downloaded direct from the site via wget to my test server...

-bash-3.00$ time php -q -f PHPCLI-LayeredCache-vbadger.php
 
1136
start loop (max = 801) (count = 1)
 #Record 1#CURRENT MEMRY USAGE = 6723868;
 
 #Record 2#CURRENT MEMRY USAGE = 9215184;
 
 #Record 3#CURRENT MEMRY USAGE = 9405564;
 
 #Record 4#CURRENT MEMRY USAGE = 9608044;
 
 #Record 5#CURRENT MEMRY USAGE = 9813064;
 
 #Record 6#CURRENT MEMRY USAGE = 10042960;
 
 #Record 7#CURRENT MEMRY USAGE = 10248712;
 
{PRESSED CTRL-C TO TERMINATE AS SCRIPT RUNNING FINE}

Chris McKee , on July 27th, 2009 at 2:50 pm Said:

ogonkov: As I've said to another person with similar issues to yourself; if your willing, email me your SSH + FTP details and site address and I'll try to resolve the problem.
Email address i on here http://chrismckee.co.uk/contact/

S , on July 27th, 2009 at 3:52 pm Said:

i download browser ver, change line $HOWTORUN = 1; to $HOWTORUN = 2 and after start get this error:
5390

ERROR: exception 'Exception' with message 'Warning: fopen(cm_cnt) [function.fopen]: failed to open stream: Permission denied in /var/www/vhosts/apolon.lt/httpdocs/badger-cm_LayeredCache.php on line 77' in /var/www/vhosts/***/httpdocs/app/code/core/Mage/Core/functions.php:239 Stack trace: #0 [internal function]: mageCoreErrorHandler(2, 'fopen(cm_cnt) [...', '/var/www/vhosts...', 77, Array) #1 /var/www/vhosts/***/httpdocs/badger-cm_LayeredCache.php(77): fopen('cm_cnt', 'w') #2 {main}

Chris McKee , on July 27th, 2009 at 4:43 pm Said:

@S: change HOWTORUN back to 1, if you read the article first you'll note that it says method 2 is for a later development.
If you still get the same error try setting (or creating two empty files of these names and setting) cm_cnt and cm_pd to CHMOD 777; as your server may restrict permissions on writing files via php.
S , on July 27th, 2009 at 5:24 pm Said:

i creating cm_cnt and cm_pd, now no error, but now get this in screen and nothing happening:
5390 start loop (max = 101) (count = 1)
after again run get this:
5391 start loop (max = 202) (count = 102)
Chris McKee , on July 27th, 2009 at 6:16 pm Said:

@S: The script will continue to run in the background; from what I can see with the error message you showed previously your running the layered cache file direct rather than the wrapper. As per the instructions open the wrapper in your browser. Not the codebehind. Let me know how you get on, my emails on the contact page if you get stuck.
S , on July 27th, 2009 at 9:22 pm Said:

Now everything fine, complete. If i want use cron job run with wget or lynx, i need run PHPCLI Layered Cache or browser Layered Cache ?
Thanks for nice script!
Chris McKee , on July 27th, 2009 at 9:55 pm Said:
If you want to use a cron job you can use the PHPCLI version.
Daniel Craig (wearebeef.co.uk) used this...

running the following task:
```
0 */3 * * * root nice -n 5 php -qf /var/www/reindex_nav.sh
```
reindex_nav.sh
```
#!/bin/bash
 
nice -n 5 php -f ./update_layered_nav.php
while [ -e ./cnt ]
do
  	nice -n 5 php -f ./update_layered_nav.php
done
exit 0
```
Oleksiy , on July 28th, 2009 at 4:07 am Said:

Thanks for the solution! However it doesn't work for me. I've used the browser version. It processed about 8K records but I still don't see layered navigation. And both catalogindex_eav and catalogindex_minimal_price tables are still empty. Fixed by update 20090728 Browser Cache Refresh; PHPCLI edition wont work on Hostmonster.com due to restrictions
Chris McKee , on July 28th, 2009 at 9:04 am Said:

@Oleksiy: If your still having difficulties feel free to email me your details and I'll try and get it up and running.
ogonkov , on July 28th, 2009 at 9:14 am Said:

Thanks for help, Chris! How did you script count total products? I see more products in admin, than script is count.
ogonkov , on July 28th, 2009 at 9:16 am Said:

I play with browser version now, and it's work without errors for me.
Chris McKee , on July 28th, 2009 at 9:28 am Said:
@ogonkov: A lot of the issues regarding running PHP direct from Bash is down to server setups varying (massivley) and various security restrictions and some just dont have zend installed which causes issues.

The product counts as follows
```
$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
$var  = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
```
Which basically says get all ID's from catalog/product collection; we then calculate the size of the array (-1 to account for the array starting from zero) and use that number to grab the record value for the last item in the array (aka the last product) We have to grab the highest product number rather than the number of products because the simple re-index processes based on product numbers.

For instance if I had 300 products; I deleted 100 out of that and added another hundred, there would still be 300 products but the ID's would of reached 400.
So if we then processed incrementally though to 300 we would actually miss a hundred records; thus if we grab the highest ID and use that as our processing target, yes we cover some blanks (which takes no real time), but we get to our actual target and therefore process all the files.

Congrats on getting it up and running )
ogonkov , on July 28th, 2009 at 10:51 am Said:

Process are end, but what i need to do next? Now i don't see any layered navigation at all
Chris McKee , on July 28th, 2009 at 11:17 am Said:

@ogonkov: Doesnt sound quite right; is your Store ID right? & is you catalogindex_eav table populated.
ogonkov , on July 28th, 2009 at 11:29 am Said:

catalogindex_eav table are empty
Store ID seems right.
I should fix catalogindex_eav table? What i need for this?
Chris McKee , on July 28th, 2009 at 12:35 pm Said:

@ogonkov: feel free to email me (tad easier on the webpage)
ogonkov , on July 28th, 2009 at 1:47 pm Said:

Drop you a letter, thanks for help, Chris!
Chris McKee , on July 28th, 2009 at 7:48 pm Said:

Latest update resolves most peoples problems: Clearing Cache Flag was failing due to the new script counting from 1 rather then 0. Please Re-download.

I'd love to see how many products people are processing with this; which version; and how long (ish) it takes

Thanks
Will , on July 29th, 2009 at 4:19 am Said:

Part Badger, Part Duck, All Delicious!

Browser version worked flawlessly with v.1.3.2.2.

There's still one thing I'm having trouble with: every time I import products via Dataflow, the layered nav is completely killed and I have to run the Badger just to get it back up. Is that meant to be the case? I thought unchecking the 'Layered Nav Cache Control Beta' box in Admin was meant to prevent this.

Anyone know how to update via Dataflow without killing Layered Nav?

Thanks again Chris for the great script!
Chris McKee , on July 29th, 2009 at 9:12 am Said:

@Will: Good stuff, If you add products via magento admin the layered nav 'should' be updated. Using the API to my recollections doesn't bother and you'll need to run the badger again.

You can of course cheat if you know where the records start. Simply take the product ID of the first new product added (minus one) and drop that in the cm_cnt file and the script will run from that point in.

Chris
E. , on July 29th, 2009 at 9:27 am Said:

Hi Chris,

Thanks for the hard work, I'm currently trying your shh version out. I just wonder, how do you know to which amount you can change the $HM variable? What happens if it's to high? I just altered it at my script to 10000 (however we have currently only 8000 products in store), but the script does seem to run fine.
Chris McKee , on July 29th, 2009 at 9:37 am Said:

@E: If the number is too high the script will eventually run out of memory and stop. When it stops just restart it.
It will cause absolutely no real issues as the persistent count and limit are stored in cm_cnt and cm_pd till the script count = total anyway. The number I put it was just an arbitrator number plucked out of the air that reached close to the max my shared host test setup would handle. (128meg Max for a php script).

If you've got the memory you could run it in batches of 1000; I cant see it completing in one run without running out of memory trying to run 10000. But as above, it wont cause an issue.

S , on July 29th, 2009 at 11:25 am Said:

Hi, i try like you write to start from cron job
1. write in cron:

 nice -n 5 php -qf var/www/vhosts/httpdocs/reindex_nav.sh

2. create new file reindex_nav.sh with:

#!/bin/bash
 
nice -n 5 php -f ./update_layered_nav.php
while [ -e ./cnt ]
do
  	nice -n 5 php -f ./update_layered_nav.php
done
exit 0

3. rename cm_LayeredCache_badger.php to update_layered_nav.php
and after run get cron email just with this:

nice -n 5 php -f ./update_layered_nav.php
while [ -e ./cnt ]
do
       nice -n 5 php -f ./update_layered_nav.php
done
exit 0

i dont know we the problem, maybe can help?

after i try in cron write like same for ssh:

time php -q -f /var/www/vhosts/httpdocs/cm_LayeredCache_badger.php

and get cron mail this:

ERROR: exception 'Exception' with message 'Warning: fopen(cm_pd): failed to open stream: Permission denied  in /var/www/vhosts/httpdocs/cm_LayeredCache_badger.php on line 40' in /var/www/vhosts/httpdocs/app/code/core/Mage/Core/functions.php:239
Stack trace:
#0 [internal function]: mageCoreErrorHandler(2, 'fopen(cm_pd): f...', '/var/www/vhosts...', 40, Array)
#1 /var/www/vhosts/httpdocs/cm_LayeredCache_badger.php(40): fopen('cm_pd', 'w')
#2 {main}
real    0m0.195s
user    0m0.106s
sys     0m0.069s

Chris McKee , on July 29th, 2009 at 11:37 am Said:

@S: It looks like a permissions issue, Warning: fopen(cm_pd): failed to open stream: Permission denied usually means that the user CRON or Apache is using isn't the same or in the group required to run that script and access that file.
You could always try SU cron; then running the function that way to see if it has the same issue.

Unfortunately one of the joys of the linux os )
S , on July 31st, 2009 at 2:08 pm Said:

I try start from ssh and work perfect, i try with default client login(same like cron) and try with root, when i try with root i found cm_pd and cm_cnt in /root/ folder, i don't thing its correct, maybe need in script write full where file locate cm_pd and cm_cnd like /var/www/vhosts/httpdocs/cm_pd i dont know php, how i can do this?
S , on July 31st, 2009 at 2:56 pm Said:

i change $prodCount = "cm_pd" and $countFile = "cm_cnt" to
$countFile = "/var/www/vhosts/httpdocs/cm_cnt"
$prodCount = "/var/www/vhosts/httpdocs/cm_pd"
and now file create without problem and error. What about $HM = 800; when run from ssh her need write max value how many product i have? I have 4500product, when her write default 500 i see in ssh like max execute 500. If i write 10000 script execute all my product.
Chris McKee , on August 5th, 2009 at 11:38 am Said:

You can change the max value to what ever you like; but if you run out of memory the script will terminate; restarting the script will then continue from where it left off.
If your using this script theres a good chance that you cant run processes for long enough to achieve running the entire catalogue in one go; otherwise the one built into magento would do the job.

I'd usually advise running the script manually starting at a high number (say 700 for 128 ram) and seeing if it falls over; if it dies lower the number if it runs successfully you can try increasing it.

Chris
S , on August 7th, 2009 at 12:05 pm Said:

Thanks, work nice with max value 10000, i have 512ram
Chris McKee , on August 7th, 2009 at 3:18 pm Said:

Updated Script Confirmed to fix Ogonkov (Лёша Огоньков) Probem
Luke Berg , on August 7th, 2009 at 5:36 pm Said:

Chris, thanks so much for your work on this and for making it available. I did my best to read through all the comments on all three posts, and I have a basic question that I couldn't find an answer to anywhere else.

We're running Magento 1.2.1, and started having problems with the layered navigation. Under 'Cache Control (beta)' we had the 'Layered Navigation' box checked, and every day, the layered navigation breaks. When I go into the cache control section of the admin, next to the "Layered Navigation Indices" there's a button that says the indexing is running, but of course it never finishes. To fix it, we've been clicking the button next to it "refresh now" and that asks if we want to kill the current process and start a new one. We do, and then it works until the next day.

From what I understand from reading your posts and the comments, unchecking the "Layered Navigation" under "Cache Control (beta)" should stop that daily refresh. Is that correct? Is that all that setting does? Or more clearly, if I turn off that setting, but go in daily and manually hit the 'Refresh now', will that do the same thing?

Is your script intended to fix the 'auto refresh' of the indices, or is it for those for whom hitting "Refresh Now" doesn't even work?

Thanks for your help!
Chris McKee , on August 7th, 2009 at 7:32 pm Said:

@Luke Berg The script is a TOTAL replacement for the admin layered navigation.
If you Got to the ‘Cache Control (beta)’ and turn off the layered navigation then run my script. The first thing the script will do is clear the flags that say the process is still running.

The problem is the amount of time to process records vs the max-execution time for the script and the amount of memory leaked.

Soooo if your on a 4GIG ECC Ram, Multicore, dedicated box you can probably get away with running the built in script with a few php.ini edits.

Otherwise disable it and use mine )
Luke Berg , on August 7th, 2009 at 10:58 pm Said:

Sorry, I don't think that answered my question, or if it did, I misunderstood it. :-\

Unchecking the box "Layered Navigation" under "Cache Control (beta)" does not turn off layered navigation, because I still see it working on my store, and I can refresh it manually by clicking 'refresh now' next to 'layered navigation indices'. What I'm guessing it does (but can't seem to confirm anywhere) is automatically do exactly what clicking 'refresh now' does.. just updates the layered navigation indices.

For us, the automatic daily procedure fails, but I'm able to do it manually.

My questions are, does that check box do more than I described above, that you know of?

Is your script intended to be run automatically on a schedule or manually?

Thanks!
Chris McKee , on August 8th, 2009 at 12:00 am Said:

It 'attempts' to run exactly the same process yes.
The 'browser' version of this script is more for people on shared hosting (though anyone can use it)
The PHP-CLI Version can be run on a cronjob.
Bill Nash , on August 15th, 2009 at 7:33 pm Said:

Chris,

I downloaded Browser Layered Cache – vbadger – v1.3.4 and uploaded and ran it. It ran through 4100 files and finished with a screen saying Files Deleted on the url badger-count.php?done=1. I read through this numerous times to see if I missed something, but I think alot of it is over my head. My admin navigation does not expand when I mouse over. Is this what your program is designed to repair? I really need help. I have been dead in the water for about a week now.

Thanks, Bill
Chris McKee , on August 15th, 2009 at 7:51 pm Said:

@Bill Nash: The script is a fix for the Cache Management - Beta Cache - Layered Navigation. As detailed in this video http://www.magentocommerce.com/media/screencasts/layerednavigation/view .

The files it deletes is just the two temporary files (counter and total).

This won't fix your Admin Navigation error; Try disabling your plugins first to see if they're causing your admin-nav issues, and theres always the possibility of reinstalling the admin/admin theme via magento connect.
Brian Mark , on August 21st, 2009 at 4:13 am Said:

We've been having a ton of trouble with this. I can set the $HM to 100,000 to do the 81k products in our store in one pass. It completes, but most of the time we get nothing showing but price, sometimes we'll get manufacturer, sometimes nothing at all. Very frustrating.

Also, is there a way to do multiple stores at once? I can run it for each of the 7, but that takes a long, long time. It's over 2 hours per store as is on a dual dual-core box with 8 gigs of RAM for both web and db servers. We have 4 more web servers besides this one with similar setups, other than the one with 32 gigs of RAM that we also use for memcached.

Anyway, what happens if I just remove that $STOREID variable? Will it do all of them at once? I've got a few hundred thousand products to add yet, so I'm sure I'll need to do some tweaking to get this working when we're all done.
Chris McKee , on August 21st, 2009 at 8:58 am Said:

@BrianMark if you remove the store id from the main function in the loop, the singleton would in theory run through the product id on all stores. Give it ago and let me know.

Oh and JESUS THATS A NICE SETUP p
Gabriel Perez , on August 21st, 2009 at 12:16 pm Said:
I found a error in the line number 99 in the CLI version.

Original:

getStore('1');

Fix:

getStore( $STOREID );

Thanks for your script!
Chris McKee , on August 21st, 2009 at 12:23 pm Said:

@Gabriel Thanks I'll changed that in a mo or so. Testing code left in me thinks!

Fixed in PHPCLI + Browser edition, issue will only affect people with multiple stores. (or stores whose ID is not 1).

Teach me to write entire scripts in SSH/VI p
Brian Mark , on August 22nd, 2009 at 12:33 am Said:

Wow. Timely fix. It's working great now that line 99 is correct. Thanks, @Gabriel.
Brian Mark , on August 22nd, 2009 at 4:09 am Said:

BTW, removing the store id just causes an error. I might have to create a loop so it'll do all 7 stores (and counting, right?) in one pass.

It'd be nice if the script could only do the products in a store instead of all, though. We've got one store that will have 600,000 products, one that has 100, one with 2,500, and one with 15,000. Doing all 600k+ for each store is very inefficient. Any thoughts there?

Thanks for the great script.
Chris McKee , on August 22nd, 2009 at 12:12 pm Said:

@BrianMark where I count the array size of Products to get the max-ID you could 'technically' go on to use that array to process just the records that are there. The only downside to this is you lose the ability for the script to fall over or be stopped and restarted from its previous place.

Really to do multiple in one run you need to add a new count file (for store id) and loop the whole script; that way the script will run the 600K for the first store, wipe the count files and move to running just the 100 for the second and so on. Shouldn't be too hard to write really but I don't have test-bases with multiple stores to whack out the changes I'm afraid.
Fred , on August 22nd, 2009 at 9:52 pm Said:

Chris,

I have the wrapper version of Badger 1.3.5 installed on a test site running Magento 1.3.2.2 on a dedicated Lightspeed server with dual CPUs and 32GB of RAM. My store has about 62,000 grouped and simple products with 109 Magento attributes that have been imported through a customized Magento import profile. The issue is Badger runs successfully (no errors), creates cm_cnt and cm_pd files and deletes the files successfully after I click the link but layered navigation does not change from before I ran the script. I have run the script several times with no results and no errors. I'm guessing there is one instruction you mentioned somewhere that I am missing.

Note: After the script runs, both files only contain the text "108988" before I delete them. This does match the count I saw on the html page as it was counting down. I appreciate the help. - Fred
James Allwood , on August 25th, 2009 at 8:36 am Said:

Hi Chris. Just noticed this update. Runs fine on my server, but on another server I'm helping with whilst trying to run the web version I get this error -

A file permissions error has occurred. Please check the permissions on the script and the directory it is in and try again.

All files are set to 777. Any ideas? Thanks
Chris McKee , on August 25th, 2009 at 9:22 am Said:

@Fred: Hard to say really; try looking in your Cache management options in Magento Admin and see if it still says that the layered cache is still running. The script should clear this flag automatically so it shouldn't, but if it hasnt the index wont update.

@James: The PHP scripts should be set to 755; if you monitor the folder you should see the cnt pd file created. If it isnt you may need to pre-create these two files (empty files) and set them to 777. Some security settings are known to cause problems with file creation.
James Allwood , on August 25th, 2009 at 9:25 am Said:

Ah, sorry problem solved. This particular server would not allow scripts with 777 to run, don't know why. Anyway, great job Chris, you're the man!
Segomel , on August 25th, 2009 at 5:21 pm Said:

hi Chris, firstly thanks for all you have done to the Magento community.
Now I wanted to let you know that the script ver that I have donwloaded from this site: http://jksite.com/blog/2009/07/magento-layered-navigation-cache-error didn’t work for me, I don’t know why!
It is connecting to the DB but once the layout comes out it says this, try it by yourself: http://nationaltree.com/mag_store/cache.php
and nothing happens, nothing changes!
Any idea why?
I also would like to know how to download this 3rd version in a zip file, if possible!
Thanks again!
Chris McKee , on August 25th, 2009 at 7:20 pm Said:

@Segemel: Not too sure as the one that JK branched from was the Part Deux script and uses none-mage-api connections to the database.

The second thing I can help you with though, the PHPCLI version zipped
Zipped PHP File
This version is to be run from SHELL or via SSH.

If you dont have access to that use the Badger-wrapper which is already zipped.
Segomel , on August 25th, 2009 at 8:25 pm Said:

Thanks budy!!
It made my Size sort list to be shown, now just 1 more thing
Take a look to my website on Christmas Trees Tab, you'll see that my Size and Price sortting list is not using all my product comparing it to my category sort list.
Do you have an idea why is that happening, weird ha?
thanks in advance!!
Fred , on August 26th, 2009 at 4:35 am Said:

Chris,

Thanks for the feedback. It does appear that the script is unable to stop the Magento refresh process once it is started. Even though I had unchecked the beta layered nav. refresh box before uploading products, once they were imported, Magento started the refresh, I started Badger which was unable to stop Magento from running the refresh process. The count on the script kept bouncing from approx. 124,000 to 102,000 back up to 110,000 but never got below 102,000. I'm guessing this was because it couldn't stop the Magento refresh.

Should the PHPCLI version potentially behave any different? Any other suggestions? Thank you.
Chris McKee , on August 26th, 2009 at 9:27 am Said:

@Segomel Not sure tbh I'll have a poke later.

@Fred Turning it off in the admin should be enough, if that isnt working. You can try disabling it as per -
http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error/
It might be running in your CRON Job, so thats worth a check too.
jonesy , on September 2nd, 2009 at 3:16 pm Said:

chris,
thanks for your great script, i've used it successfully on my shop via browser (i got the "files deleted" screen). however i still can't get the prices on grouped objects in layered navigation to show! i'm banging my head to no avail. do you have any suggestions?
Brian Mark , on September 3rd, 2009 at 5:35 am Said:

I've noticed that navigation categories don't show up in search. Any idea what sets that?
len3121 , on September 3rd, 2009 at 8:54 am Said:

Hi Chris, thanks for this great script. I am using the browser version of your script and am on Magento 1.3.2.1. I clear the cache and any flags in the core_flag table. Everything is turned off in the cache are on the backend of Magento. I then run the wrapper and everything seems to go perfectly in the browser. However after multiple attempts and checking I see that as soon as the wrapper is started a catalogindex flag pops up in the core_flag table. So although the script goes through all the products when I look at the layered nav on my site many of my custom attributes are incorrect by a wide margin. Price, category are correct but my custom attributes have only 0-50 products showing when it should be closer to 3500. Any ideas on why this (catalogindex popping up in core_flag as soon as script is started) would be happening and what I can do to fix it? Does having flat category or flat product enabled have any negative effects on the script? Thanks for your help.
Chris McKee , on September 4th, 2009 at 10:09 am Said:

@jonesy not quite sure there's the following main singleton methods related to cache;
* Mage::getSingleton('catalogindex/indexer')->plainReindex(); - Layered Nav
* Mage::getSingleton('catalogsearch/fulltext')->rebuildIndex(); - Search Index
* Mage::getSingleton('cataloginventory/stock_status')->rebuild(); - Stock Cache
* Mage::getSingleton('catalog/url')->refreshRewrites(); - Catalogue (URL) Cache

Theres probably a few more since the inception of new beta-cache methods in the more recent versions so I'll have to check.

@Brian Mark: Not sure what you mean; email me and send a screen shot my heads full of all sorts at the moment.

@len3121 The main problem that causes the flag to reappear (as noted by Fred) was Magento itself; If a product is updated / imported / added then the layered navigation will be kicked off again.
Cant really say anything about the flat-category/product options yet but that could be causing an issue; try regenerating the flat files post-index. I'll have to look into this later so if this method works for you let me know and I'll look into combining the functionality.

Chris
Brian Mark , on September 7th, 2009 at 3:42 am Said:

Ok, so it had to do with making our first level categories no longer anchors. If something matched there, we only showed 1 under the main category instead of how many were in categories under that. Made it pretty useless. All better now, although I don't like categories with 2000+ matches.
salo , on September 8th, 2009 at 4:12 pm Said:

It's a Miracle!
You save my time!
just fix $sx = Mage::app()->getStore('1'); to $sx = Mage::app()->getStore($STOREID);
daniel , on September 11th, 2009 at 2:54 pm Said:

Hi. Ran the script and it runs fine. I have a store that is used for development
real 2m1.434s
user 0m32.788s
sys 0m0.637s

Is the script designed to solve the following problem.
1. Navigate to From admin -> Catalog -> Manage Categories,
2. The categories show the wrong number of products in the categories!
3 For example, a category 'Pets' which has only 4 products, yet the category management page lists 130.

Has anyone encountered a similar issue? Or know how to go about troubleshooting?
Chris McKee , on September 14th, 2009 at 10:43 am Said:

@daniel yours sounds like a glitch in counting on the admin-catergory management which is an odd one; have you tried a) clearing all cache and b) adding/removing a product to see if it affects the count.

Chris
N Lopez , on September 14th, 2009 at 5:28 pm Said:

Thanks for the script, I am using the Browser Edition and I have a problem with the ID count, I have my highest ID is 19,833 and the script keeps counting only 16,710. I added about 100 more products and script keeps giving the same 16,710 count.
Do you have an idea what the problem can be?

Thank You
Chris McKee , on September 14th, 2009 at 6:19 pm Said:

@N Lopez: check + delete the cm_cnt cm_pd files thatll probably solve it
N Lopez , on September 14th, 2009 at 7:53 pm Said:

The files are been deleted properly after each script termination. What I found looking at my product listing was the product # 16709 was the only one I had with multiple select attributes, so I deleted it to see if that was the problem but the ID count went even lower to 16371. I am looking at the products around that ID but there is nothing unusual.
Chris McKee , on September 14th, 2009 at 8:09 pm Said:

@N Lopez: The "count" is the final product ID in the product-array. If you believe its wrong you can manually create cm_pd (the total) prior to starting the script, literally just stick in the number you think it is.
Kris , on September 18th, 2009 at 4:52 am Said:

I started to use the browser version, and it started to work and went through all the 6000 products but the indices only updated through like the first 2000k products. I have "Layered Navigation" unchecked. The one thing I did notice though is that "Layered Navigation Indices" still has "running... kill" next to it.
Anton , on September 18th, 2009 at 10:02 am Said:

as far as I know magento rebuilds Layered Navigation for all products after import, not only for that were imported. How do you think may be it will be better to rebuild only for that were imported?
Kris , on September 18th, 2009 at 7:49 pm Said:

Hey its me again. I gave this page to my host (I told him the browser version was only working for half of my products) and he did the CLI version of this test and it worked perfectly. My question is I've read this article (and the previous two) and I can't figure out how to correctly run it. I've never setup a cron job before. Now don't get me wrong, I know CSS, HTML and all that good stuff, but I would say I am new to database management. Could someone explain the proper steps to do the CLI version to me? Thanks
Chris McKee , on September 18th, 2009 at 10:53 pm Said:

@Anton all automatic indexing is disable-able; but ye its a pain in the arse, if you stop all the automatic indexing you can use the script on its own. Of course I'm going to disclaim here that I don't advise the manual editing (rather than over-riding) of Mage Core scripts.

@Kris Whos your host?
As for the CRON job your host should be able to give you instructions but here's a generated script to for a daily run...
http://www.webmaster-toolkit.com/cron-generator.shtml?minute=30&hour=23&day=&month=&weekday=&script=php+-q+-f+/rootto/the/badgerscript.php
Kris , on September 19th, 2009 at 2:16 am Said:

@chris

it is nexcess.net

So I should just put that script in my magentos root folder?
Chris McKee , on September 19th, 2009 at 12:00 pm Said:

@Kris Nice choice of hosting, one of the bigger beasts of Magento Hosting. The Badger CLI script should go into the root folder of your site; I Imagine you have a control panel system on Nexcess so theres probably a section in that for managing cron jobs. I grabbed this link from the Nexcess Knowledge Base http://interworx.com/support/docs/iworx-cp/webadmin/cron-jobs/howto-add-cron-job
Paul , on September 23rd, 2009 at 1:56 pm Said:

Hi,

Is it supposed to run for a very long time? It's been refreshing every 5 sec. for about 15 minutes now;-)

Hopefully this is correct and it will end a year of troubling layered nav
Chris McKee , on September 23rd, 2009 at 1:59 pm Said:

@Paul yep time depends on the number of products + attributes; Every 5 seconds seems slow so you either have a lot of attributes or a slow Database Setup; Normally it takes a second or less per product.
Chris McKee , on September 23rd, 2009 at 2:58 pm Said:

@Paul: Have the "count" files been created? If not you may need to do it manually; accessing badger-count.php id expect to see the output of the current progress (less you've deleted them already); if not does your site have error-output turned on?

Try running the CLI version from SSH...
Paul , on September 23rd, 2009 at 7:11 pm Said:

I have 2 cnt files. 1 cnt, 1 cm_cnt. The first one is possibly from your old version;-)
The cm_cnt is created today and contains the number 1300-something, which would be about right.
I'd love to have the browser version working instead of messing around with Putty
Anything we might have missed?
-------------
Ok, i copied cn_cnt and renamed it cnt_pd, it worked fine i guess.
[Count: 1317 | Total: 1317][ 0 to go]
Click Here To Complete and Delete Count files

But... maybe i don't get it, i still don't get any of the layered nav options other than Category: http://www.espressoshop.nl/thee.html

*slightly embarrased
Chris McKee , on September 23rd, 2009 at 8:09 pm Said:

@Paul: cm_cnt Should Start at Zero; cm_pd should be automatically created with the total of your products in it (i.e the highest ID)

Copying CM_CNT -> CM_PD would essentially say the process is complete.
Paul , on September 23rd, 2009 at 8:09 pm Said:

Screw me, it works now as it should. This made me think:
"change HOWTORUN back to 1, if you read the article first you’ll note that it says method 2 is for a later development."

* more embarrased;-)
Paul , on September 23rd, 2009 at 8:12 pm Said:

Thank you for the prompt help. Before posting i was already amazed by your involvement in the product and comments. My posts today prove nothing less than just that: commitment and pleasure in your own stuff.

Two thumbs up, keep up the good work!
Chris McKee , on September 23rd, 2009 at 8:13 pm Said:

More than happy to hear its working for you; well done and thanks, much appreciated )
Paul , on September 23rd, 2009 at 8:26 pm Said:

Nonetheless, my "category only layered nav"-feature still remains;-)

(ed- Done)
Paul , on September 26th, 2009 at 2:14 pm Said:

Chris, still no dice but i may have found why. You might have a final word on this:
I exported all products and noticed this value for each item, regardsless the store they are mainly for: store_id = 0.

In your script this is set to 1. Now i do not know if this is an important setting, but i have set it to 0 and it is runnnig now. You think that might be it? (411 to go... )
Paul , on September 26th, 2009 at 2:18 pm Said:

Nope:(
Chris McKee , on September 26th, 2009 at 3:03 pm Said:

@Paul I cant reproduce the problem in any test bases; but The storeid should match the store id you can see in the URL string in the Magento Admin (store management). Generally if you've setup a single store that ID is one.
Paul , on September 26th, 2009 at 5:25 pm Said:

Ok, we have it up to 15 up there, however i have been testing store 1 all the time. Will tryo some others too.

...

Still nothing. Too bad we have a lot of great attributes lined up for searches :/
Chris McKee , on September 26th, 2009 at 8:38 pm Said:

Emailed you; before this starts to hog the entire page ;o)
Rogier Bom , on October 1st, 2009 at 10:01 am Said:

Chris, you asked for stats, so here we go:

#Record 101#CURRENT MEMRY USAGE = 10154748;

real 13m39.059s
user 6m25.255s
sys 0m14.638s

As you can see it's much slower than your results. We're on Magento 1.3.2.4 with 15.600+ products.

Thank you for all your effort on this issue. Can't wait for this to be fixed, hopefully in 1.4
Chris McKee , on October 1st, 2009 at 10:47 am Said:

@Rogier Bom: I found that running the new code-compiler made all of this go a lot faster, of course alot depends on your setup, memory, db etc. Big queries take along time.

The script should still be useful for timeouts (which will still occur) the optimizations on 1.4 simply condense the query and make it a tad less painful for the CPU/DB/MEMORY.

Chris
Rogier Bom , on October 1st, 2009 at 11:38 am Said:

Do you have any experience with the new compiler in a production environment? I'm a bit hesitant, but with the number of products in this store things are starting to slowdown bigtime. Let's see what it does in the test environment first.
Chris McKee , on October 1st, 2009 at 11:57 am Said:

@Rogier Bom: I actually ran it on a staging server first then uploaded the whole site; not had any problems yet *touch wood* but I have had a decent performance improvement.
It "MAY" improve indexing speed but take into account the majority of the time delay on the indexer at the moment is the time between the transaction statement being created in the PHP and how long it takes to execute per product.

Optimizing your database setup to match their enterprise whitepaper is going to achieve more in the long run.
Jason hasani , on October 8th, 2009 at 10:21 pm Said:

Thanks Chris for this solution. I have a dumb question tough. After you download the files where do you copy them? I know that there's a folder called magento root but I don't know the location of that folder. I have vps hosting so I have ssh access too. Please include as much details as possible as I'm not very familiar with linux system.

Thanks in advance
Troy , on October 11th, 2009 at 8:42 pm Said:

I seem to be having some troubles with setting up the cron. The instructions you gave are perfect, but it doesn't seem to loop through the count. I have 28000 products, and occasionally it will give an our of memory error goìng through 1000 products at a time. Is there a way to get the reindex_nav.sh to loop through till its complete?

Thanks in advance!
chris wood , on October 18th, 2009 at 11:28 pm Said:

hi,

THANKS so much for spending yore time to make this script and for sharing it free of charge! i just used it to fix my LN problem and the browser version worked perfectly!

(p.s im on shared hosting if that helps!)

how do i cancel magento trying to run the indexing (so i just do it manually with the cheeky badger) and it doesnt mess up my whole nav system?

many thanks!

chris - manchester uk
Chris McKee , on October 19th, 2009 at 2:23 am Said:
@Jason hasani: just use FTP; there should be a folder called public_html / WWW or something like that (depending on the setup) which will contain the .htaccess / index.php, app folder; basically all of your magento setup; thats the root of magento.

@Troy: if you can work out how long it takes to do a "run" at a limit that doesnt burst your memory GC then you can create an extra shell script like below
```
>> FILE NAME reindex_nav.sh
#!/bin/bash

nice -n 5 php -f ./update_layered_nav.php
while [ -e ./cm_cnt]
do
  	nice -n 5 php -f ./update_layered_nav.php
done
exit 0
```
As I posted (http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-badger-part-duck/#comment-183)
Basically that should run the php file till the cnt file is destroyed
Obviously modify the script to use the file names you have (i.e. update_layered_nav.php > cm_LayeredCache_badger.php)

@chris wood: Hello from Bolton; Basically you need to turn it off in the nav... then if magento is still being a git you can nip into the indexer file and disable more bits of it. I try to avoid this as it means editing CORE files; but if its a regularly updated site then you'll find that magento tries to reindex after you add products, edit products change category details etc etc etc.

Good news is; Magento 1.4 has been optimized quite alot; and post wedding I'll be reviewing the changes in a long winded follow up to my request for data.
chris wood , on October 21st, 2009 at 5:01 pm Said:

Hey chris mckee,

thanks for the feedback. is the badger-wrapper only supposed to run once? i navigated to it a second time but it doesn't really do much..in fact it does nothing just displays this message "Layered Navigation Refresh From Browser"..

i really liked seeing it counting last time, was nice to see the progress as opposed to with the standard magento system!

thanks

chris
Chris McKee , on October 21st, 2009 at 7:37 pm Said:

@chris wood: Basically you run it and it runs 2 scripts in iframes; One that does the processing and the counter file which just shows you your progress based on the the count and final count files; the counter script also handles the bit at the end where it offers you the chance to delete the two count files and end the script.
If you had issues check to make sure that the two count files were deleted at the end; as on some setups PHP doesn't have file system delete permissions.
chris wood , on October 21st, 2009 at 8:32 pm Said:

Hi Chris McKee,

thanks for that...sorry if i'm being really dumb here but where would i find the count files that should be deleted?

i think that this should be the problem as i cant think of anything else...

thanks in advance

chris
chris wood , on October 21st, 2009 at 10:17 pm Said:

Hi Chris McKee,

ignore my last post please, I actually found out that they should be in the same folder (by running the script on a different magento installation) but they are not there so they were actually deleted when i ran the script the 1st time.

I copied fresh badger files into my original directory and it still refuses to run again the page loads but it seems like it fails to initiate, it just keeps refreshing every 10 seconds or so.

so right now it is running fine on my other install for the first time using the same files.....(this is a live post, im almost commentating as it goes!)

ok so, the file ran fine on the new directory for the 1st time, and just ran it a second time and that is fine too! confused.

so now i need to work out why it isn't working on the original directory but it is on the second directory - how frustrating that it works in my test enviro but not my live enviro!

these are things i am thinking...could it be:

1. layered nav indexing is running in the back end of my magento install? i tried to stop it but it just queues ??
2. that the directory is password protected using apache - pretty sure that this doesn't make any difference...

please help if you can, not having my layered navigation is really worrying me!

regards

chris
Chris McKee , on October 22nd, 2009 at 11:00 pm Said:

Sorry for the late reply, I'm a tad busy getting married so this will be my final reply to anything till the 9th November. K' Thnx' Bye Bye
Mike , on October 24th, 2009 at 9:59 pm Said:

Any chance of getting this to work with Magento ver. 1.4.0.0-alpha3?

Thanks
saho , on October 25th, 2009 at 12:36 am Said:

@ Brian Mark

We had the same problem where only price was showing after. It was due to a module called Lucky_InStockOnly that hides out of stock products but the creator added new code to make it work.

Anyone else who has this module and the same problem look at the updated blog entry. Took me forever to figure out what was going on.

http://colin.mollenhour.com/2009/09/yet-another-hiding-out-of-stock-products-update/
len3121 , on November 10th, 2009 at 3:19 pm Said:

Since Chris is busy getting married (congrats btw) I figured I would ask if anyone looking at this thread knows how to stop the browser version from running? i tried running it on a different store id for my subdomain but i must have made a mistake since all I see is [Count: | Total: 3392][ 3392 to go] and that never changes. I assume it is not seeing the count file and will perpetually run. Any way to stop it?
andrew , on November 12th, 2009 at 2:34 am Said:

I have used the browser version and it seems to be successful. I am trying to use this to be able to sort my products by price once again. Even after running this, the products won’t sort correctly. Do you think I am doing something wrong with this script or is it possibly another error?
Daniel , on November 12th, 2009 at 5:13 pm Said:

hey hey,

I got a problem, my navigation keeps on running.
I even changed to a complete new server with magento hosting.
i kept the same problemen, can't run it.

Now i uploaden you script, everythings seems fine but it keeps saying:
[Count: | Total: 2731][ 2731 to go]

Nothing happends? Please help!
craigcriv , on November 14th, 2009 at 9:54 pm Said:

Why do I get an error 500- internal server error when trying to run the file?

On the browser version

magento 1.32.3
Paul , on November 17th, 2009 at 2:21 pm Said:

Hi there. I LOVE your script. works wonderfully. My question is the mangento Layered Navagation Intices keeps running automatically everyday, fucking up my layered navagation. do you know how to stop it from running on its own?
Chris McKee , on November 17th, 2009 at 8:45 pm Said:

Check the file permissions, and check your apache/lighttpd ~ server error log for details about the error. Turn on PHP error mode, probably wont make a difference though; your logs are the best bet.
Chris McKee , on November 17th, 2009 at 8:50 pm Said:

@Daniel Delete the cmcnt and cmpd files; some servers dont allow php delete file operations.
Chris McKee , on November 17th, 2009 at 8:59 pm Said:

@andrew I think someone asked this one before and it was related to the new compiler. My heads a tad fuzzy at the moment so when it clears up i'll have a prod around.
Chris McKee , on November 17th, 2009 at 9:01 pm Said:

@len3121 The browser version offers an option on completion to delete the two files it creates. If you've closed the window prematurely simply delete the two counter files.
andrew , on November 18th, 2009 at 9:56 pm Said:

@Chris - Thanks for replying. Are you referring to the messages above to/from Rogier Bom? I took a look at those but wasn't able to find anything to help me.
craigcriv , on November 19th, 2009 at 10:03 am Said:

Can't access the error logs. Where would I add this script to create an error log and where would I find the file ?
I have the permissions set to 777

Creating own error logs for debugging PHP scripts.

Since we don't provide access to Apache error logs on shared hosting packages for
technical reasons, you can create your own error logs for debugging PHP Scripts.

Please insert the following code in your PHP script (or create separate file and
and add the code in it. Include the file using "include()")

error_reporting(0);
$old_error_handler = set_error_handler("userErrorHandler");

function userErrorHandler ($errno, $errmsg, $filename, $linenum, $vars)
{
$time=date("d M Y H:i:s");
// Get the error type from the error number
$errortype = array (1 => "Error",
2 => "Warning",
4 => "Parsing Error",
8 => "Notice",
16 => "Core Error",
32 => "Core Warning",
64 => "Compile Error",
128 => "Compile Warning",
256 => "User Error",
512 => "User Warning",
1024 => "User Notice");
$errlevel=$errortype[$errno];

//Write error to log file (CSV format)
$errfile=fopen("errors.csv","a");
fputs($errfile,"\"$time\",\"$filename:
$linenum\",\"($errlevel) $errmsg\"\r\n");
fclose($errfile);

if($errno!=2 && $errno!=8) {
//Terminate script if fatal error
die("A fatal error has occurred. Script execution has been aborted");
}
}
Chris McKee , on November 19th, 2009 at 10:09 am Said:

@craigcriv file should be set to 0755 (or if your using an ftp client it may only accept 3 numbers, these would be 755)

That script could be included at the top of each PHP file after the
craigcriv , on November 20th, 2009 at 9:53 am Said:

Thanks Chris Got it to work, had wrong path ( /magento/badger-wrapper.php )
Should not have added magento in the path.

I did get the layered navigation to appear but not all the manufacturers are showing up n the search by only one, and the item counts in the price or manufacturer are not correct.
Paul , on November 30th, 2009 at 3:45 am Said:

i have 2 questions. 1st how can i run this script from cron? I don't have SSH access though. and 2nd magento's builtin layered navagation indices keeps running automatically. how can i stop it? i've already removed it from the cron scheduler
Sergey Leongardt , on December 1st, 2009 at 10:54 pm Said:

We spent countless hours trying to get the Refresh of Layered Navigation Indices to work.

We had close to 10000 products on pacwave.com runnning on a VPS, and nothing we did worked.

The solution was to add RLimitCPU 1200 to the .htaccess file. This basically prevents the refresh script from consuming too much of the CPU resources and crashing.

Hope this helps some of you.
Gav , on December 2nd, 2009 at 4:51 am Said:

I've run this script successfully a few times but occassionally I get the following error. The script completes, I delete the files through the link, refresh the cache. Now I have a working front page, but all of the links from the featured products on the front page and the links in the nav bar give the following:

Not Found
The requested URL /home-electrical.html was not found on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.7a DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server at http://www.xxxxxxx.co.uk Port 80

I've managed to get the site to work again in the past by some weird combinations of rerunning the script and refreshing the cashces but this time it seems to be stuk completely! All of the items (4,900) and categories are still showing in the admin panel! Any ideas?

Cheers
Gav
multivac , on December 3rd, 2009 at 11:50 am Said:

Hí

I have a problem with grouped products. I can´t see "Starting at: ...." I solved using "Layered Navigation Indices". Now I have 2000 products and fails because "my set_time_limit" with my server is 30 seconds. I'm using your script. It runs well (i think...) but do not get the same result as doing it from the admin of magento. I can not see "Starting at: ...." For a long time I am trying to solve this problem, but can not find a solution.

Can this script help me?

Thanks a lot.
Joro , on December 14th, 2009 at 8:31 pm Said:

Hi Chris,

Your script is great, but it can refresh completely my layered nav.

I'm using just one attribute for Age in my store for almost 1K products. When I refresh the nav, can't find all available products... approximately 1/3 of them is missing.

In the past, when I added new products, they appeared immediately in to the layered nav, but since one month it' stopped. The only solution was to use your script, cause the refresh of the Indices never ends. Also, I have a price rule, that stops works in every 2 ~ 3 days and I must recreate it. First I thought, that there is a relation between, but in fact NO, when I stopped the rule, the lay nav stays the same - incomplete.

Finally, I have an amount of products that is available in the layered navigation, but when I add new products, it's impossible to show them in.

Cheers form Bulgaria
Vpas , on December 23rd, 2009 at 8:54 pm Said:

You are great!
You gave me an exceptional and professional solution. I run the browser package and boom the scripts did their work.

Thanks again. I was really stressed with this refresh problem. You helped me solve with the most elegance solution one my pressing problems so far.

Vpas
Zerone , on January 1st, 2010 at 2:52 pm Said:

The PHP-CLI and browser version work correctly, but I have a question:
In admin panel the Layered Navigation is disabled,
however I find that every day starts the rebuild layered navigation index ¿?
At the time I run your script so that everything goes well.
What is the reason to launch automatically Layered Navigation Index?
I programmed in crontab cron.php script. Does this spear?
Thanks
Chris McKee , on January 1st, 2010 at 3:35 pm Said:

There are several functions within magento that will kick-start the internal process; unfortunately these are all within the core files.

Zerone , on January 1st, 2010 at 3:48 pm Said:



 
What is the reason to launch automatically Layered Navigation Index?

I autoresponder, if anyone is interested.
See app\code\core\Mage\CatalogIndex\etc\config.xml the default values:



    .........
    .........
 
 
 
                0 2 * * *
                catalogindex/observer::reindexAll
 
 
                * * * * *
                catalogindex/observer::runQueuedIndexing
 
 
 
    ......
    ......

Thanks,

Zerone , on January 1st, 2010 at 3:50 pm Said:

Sorry,
This is the code:

0 2 * * *
catalogindex/observer::reindexAll

* * * * *
catalogindex/observer::runQueuedIndexing
Eric , on January 4th, 2010 at 1:31 am Said:

Hi Chris,

Do you know what would be the best way to stop magento from automatically trying to refresh the layered nav after a fresh import.

Basically, I assumed it had something to do with a cronjob.

As recommended by others, I commented out:



in app\code\core\Mage\CatalogIndex\etc\config.xml

But still no luck.

Any ideas, or know where to start?

Many thanks,

Eric
Gav , on January 8th, 2010 at 8:48 pm Said:

Just FYI - my problem turned out to be yet another Magento well known but as yet still unfixed bug - the URL rewrite bug. It had nothing to do with Chris's code at all
craig , on January 13th, 2010 at 6:02 am Said:

Do you have a code or app to rebuild the search index?

I have the same problem with the server timeout like I did with the layered navigation rebuild, which you fixed with the badger wrapper app.

Thanks
Nick , on January 25th, 2010 at 5:24 pm Said:

Chris - Thank you for this solution. I have the browser version running on my 9,000 test products and everything appears to be working correctly (script says its completed and I delete the count files). However, when I go to a category that I have set to anchor the layered navigation count does not match the number of products.

Example:

Category A has 47 products - I can browse through all of them without issue.

Layered Navigaton has the following:
$0.00 - $100.00 (5)
$100.00 - $200.00 (1)
$200.00 - $300.00 (1)

Any reason you konw of why this would be happening? I treid to rebuild the other indexes and that didn't seem to fix it. I have not run a product load since I began testing this script.

Thanks ahead of time!
Res , on February 2nd, 2010 at 6:28 pm Said:

Hi Chris,

a well done script and so necessary as we all need it!

In my case it does not work correctly. Take a look at yourself: http://www.brady-schweiz.ch/badger-wrapper.php

I assume this is due to several storeIDs in this site: 0 for admin and 1,2,3 for three languages.

It starts with 27803 records, but I don't have so many products, only about 16000.

Could it be, that this count is the total without regard to the storeID?

I would pay you the hours for adjusting this script to work with several storeIDs.
These should be gotten from the shop itself or setup by an array.

Do you have time for this?

yours
Res
MagicCardz , on February 7th, 2010 at 2:20 pm Said:

You are a Genius!
You "Magento Layered Cache Fix Browser Edition" work perfectly with my share hosting.

Thanks!!
Josh , on February 11th, 2010 at 4:51 pm Said:

Hi Chris this script is great, saved me a lot of problems.

I have one question that I'm hoping you might know the answer to, in our store we have reached now around 2900 products, your script was working fine up till about 2400 but recently when I run it it thinks that it has done all the products but the last 400 or so don't get refreshed. Any ideas how I can fix this?

Many thanks
Josh
Chris McKee , on February 11th, 2010 at 9:43 pm Said:

Magento 1.4 solves many of these issues (zend caching & mage improvements)
http://twitter.com/YoavMagento/status/8974788383
So I'd say hang on for a mo and test it out (see if it runs it all).

The Script May Not Run on Magento 1.4 (as the issue is 1.3 related) If people still have this issue in 1.4 I will redevelop it; just let me know if you still have issues)

Thanks

Chris
Lena , on February 26th, 2010 at 3:39 am Said:

Thanks for the nice post. It was a really good read. Hope to read more of your articles.
Slack , on March 4th, 2010 at 6:15 am Said:

Hi Chris.

I got Magento 1.3.1 and I made an upgrade (using Magento Connect) to 1.3.2.4.
Since that upgrade, layered navigation works but I can't see the price filter.
I've executed your script, but nothing change on the layered navigation.
Any idea?

Cheers.
Chris McKee , on March 4th, 2010 at 9:31 am Said:

@Slack: at this juncture I'd advise upgrading to 1.4.1 to escape the 1.3 range of issues. It fixes quite alot.
Slack , on March 5th, 2010 at 2:39 am Said:

Hi Cris.
Thank you for your response.

» Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck | Chris Mckee Designs, by Chris McKee, who'd of guessed | ChrisMcKee.co.uk