Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck

Solution 3 – Running the Layered Navigation Cache; PHP-CLI & Browser Version

Magento 1.4 will probably solve this issue for many people, so it may be worth giving it a squirt :o)

Fuck up central, exporting the script caused some of the comments to drop to multiple lines. If you had problems re-download please.
Browser edition now updated to actually delete files on completion and in clicking complete deleting the files and stopping the refreshes. :)
———————————-
Ok so I buggered up the first release (hence the post dissapearing), Basically I’d “counted” the products rather than grabbing the last ProductID as used by the Singleton method. So I’ve resolved that (thanks Mark H for pointing it out, I must of been half asleep). Processing through none existant records has no detrimental effect (it checks and jumps them) so I’ll skip on the suggestion of running through the array, thanks.

Well here goes, part three… Deep Breath.

First some specs:

On my test server (restrictions as on shared hosting in CHROOT);
100 records took this long

Previous 100 records in

#Record 101#CURRENT MEMRY USAGE = 152,411,480;
real    5m37.504s
user    1m19.869s
sys     0m3.362s

New 100 records in

#Record 101#CURRENT MEMRY USAGE = 40,411,480;
real    2m30.101s
user    0m34.841s
sys     0m0.907s

New Features

*NEW* PHP-CLI Edition
* Auto Product Count
* Auto On Completion Destroy Temp Files
* Various Optimizations

*NEW* PHP Web Browser Edition
* Three files;
** Wrapper (which you run from your browser)
** Two IFRAMES: The main functional script and a counter to show its not fallen asleep
** Auto Updating output; set off and sleep mode :op

 


Now for the breakdown (script not mental)

 

Magento Layered Cache Fix PHP-CLI Edition

#!/usr/bin/php -q
<?php
//#######################################
//# Magento Layered Nav - Badger || Duck
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# MLBv3.1.1
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
try{
	ob_implicit_flush(true); //Saves having to flush manually
	if(!ini_get('safe_mode')){set_time_limit(0);}/*Set time limit to unlimited, though we shouldnt need to*/
	ignore_user_abort(); //Lets the script run to completion even if you kill the browser
 
	require_once("app/Mage.php");
	Mage::app('admin'); //as per magento handling
	Mage::setIsDeveloperMode(true); //apmh
	umask(0);
 
	/*{ CONFIGURABLES }*/
	$HM = 800; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
 
	/* Automatically Count Products */
		$total = NULL;
		if($HOWTORUN == 1)
		{
			/*## CLI Version */
			$prodCount = "cm_pd";
			if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
			else
			{
					$fh = fopen($prodCount, 'w') or die("can't open file");
 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
 
					$total = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
 
					fwrite($fh, $total);
					fclose($fh);
			}
			/*## End CLI Version*/
		}
		else
		{
			/*## Browser Version */
			if((!isset($_SESSION['pcount']))||($_SESSION['pcount']==NULL)){ 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$endValue[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
					$_SESSION['pcount'] = $endValue;
				$total = $_SESSION['pcount'];
			}
			else{ $total = $_SESSION['pcount']; }
			/*## End Browser Version */
		}
		if($total == NULL){ throw new Exception("Total Wasn't Captured"); }
	/* End Count Products */
 
	echo("n$totaln");
 
	$countFile = "cm_cnt";
	$count = null;
	$store = Mage::app()->getStore($STOREID);
 
	//Create a count file if we dont have one
	if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	else
	{
			$fh = fopen($countFile, 'w') or die("can't open file");
			$count = 0;
			if((int)$count == 0) $count = 1;
			fwrite($fh, $count);
			fclose($fh);
	}
 
	//If we havent started counting yet clear cache.
 
	if($count < 2)
	{
			Mage::app()->cleanCache();
			//Clear the database flag from any previous "normal" indexes that failed
			$flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
			if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) $flag->delete();
	}
 
	$max = ($count+$HM);
	$sx = Mage::app()->getStore('1');
 
	ob_start();
	echo "start loop (max = $max) (count = $count)";
 
	for($i = $count; $i <= $max; $i++)
	{
	ob_end_flush();
	ob_start();
				if($i == $total)
				{
						echo "<br><h1>Completed</h1><br>";
						@unlink($prodCount);
						@unlink($countFile);
					exit(1);
				}
 
 
			Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $sx);
 
			echo("n #Record $i#CURRENT MEMRY USAGE = ".memory_get_usage().";n"); //UPDATE SCREEN
 
			//Write where were up to
			$fh = fopen($countFile, 'w') or die("can't open file");
			fwrite($fh, $i);
			fclose($fh);
	ob_flush();
	flush();
	}
 
}catch(Exception $e){ echo"<br><p style=red>ERROR: $e</p><br>"; exit(1);}
#EOF
?>

Mostly though you’ll need to change this bit to run it in PHP-CLI over SSH…

/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
  • How many records to process at once: I comfortably managed 500 at a time without it breaking a sweat
  • StoreID: I’ll let you guess, but for most of us mortals 1 is the default.
  • HowToRun: This is extendable so basically the guts is there to allow you to use Sessions (in the browser version) IO readings slightly faster so in the end it won over sessions

Thats pretty much it, aha, yep thats all, stop asking.

Simply run it from shell (or a script) as follows

time php -q -f cm_LayeredCache_badger.php

Time will output the time it took to run the script; -f is telling php to load the file; -q is quiet mode, but thats on by default.

Magento Layered Cache Fix Browser Edition

badger-cm_LayeredCache.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
try{
	ob_implicit_flush(true); //Saves having to flush manually
	if(!ini_get('safe_mode')){set_time_limit(0);}/*Set time limit to unlimited, though we shouldnt need to*/
	ignore_user_abort(); //Lets the script run to completion even if you kill the browser
 
	require_once("app/Mage.php");
	Mage::app('admin'); //as per magento handling
	Mage::setIsDeveloperMode(true); //apmh
	umask(0);
 
	/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
 
	/* Automatically Count Products */
		$total = NULL;
		if($HOWTORUN == 1)
		{
			/*## CLI Version */
			$prodCount = "cm_pd";
			if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
			else
			{
					$fh = fopen($prodCount, 'w') or die("can't open file");
 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$total = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
 
					fwrite($fh, $total);
					fclose($fh);
			}
			/*## End CLI Version*/
		}
		else
		{
			/*## Browser Version */
			if((!isset($_SESSION['pcount']))||($_SESSION['pcount']==NULL)){ 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$_SESSION['pcount'] = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
				$total = $_SESSION['pcount'];
			}
			else{ $total = $_SESSION['pcount']; }
			/*## End Browser Version */
		}
		if($total == NULL){ throw new Exception("Total Wasn't Captured"); }
	/* End Count Products */
 
	echo("n$totaln");
 
	$countFile = "cm_cnt";
	$count = null;
	$store = Mage::app()->getStore($STOREID);
 
	//Create a count file if we dont have one
	if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	else
	{
			$fh = fopen($countFile, 'w') or die("can't open file");
			$count = 0;
			if((int)$count == 0) $count = 1;
			fwrite($fh, $count);
			fclose($fh);
	}
 
	//If we havent started counting yet clear cache.
 
	if($count < 2)
	{
			Mage::app()->cleanCache();
			//Clear the database flag from any previous "normal" indexes that failed
			$flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
			if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) $flag->delete();
	}
 
	$max = ($count+$HM);
	$sx = Mage::app()->getStore('1');
 
	ob_start();
	echo "start loop (max = $max) (count = $count)";
 
	for($i = $count; $i <= $max; $i++)
	{
	ob_end_flush();
	ob_start();
				if($i == $total)
				{
					echo "<br><h1>Completed</h1><br>";
					exit(1);
				}
			Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $sx);
 
			//Write where were up to
			$fh = fopen($countFile, 'w') or die("can't open file");
			fwrite($fh, $i);
			fclose($fh);
	ob_flush();
	flush();
	}
 
}catch(Exception $e){ echo"<br><p style=red>ERROR: $e</p><br>"; exit(1);}
#EOF
?>


badger-count.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck - Counter
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
?>
<html>
<head>
<?php if(!isset($_GET["done"])){ ?>
<meta http-equiv="refresh" content="5" />
<?php } ?>
</head>
<body>
<?php
$prodCount = "cm_pd";
$countFile = "cm_cnt";
 
 
if(isset($_GET["done"])){ 
	unlink($prodCount); 
	unlink($countFile);
	echo "Files Deleted";
	exit(1);
}
 
if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	if(isset($total)){
		$left = ($total - $count);
		echo "<b>[Count: $count | Total: $total][ $left to go]</b>";
				if($left == 0)
				{
					echo "<br /><a href='badger-count.php?done=1' target='_parent'>Click Here To Complete and Delete Count files</a>";
					exit(1);
				}
	}
?>
</body>
</html>


badger-wrapper.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck - WRAPPER
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-gb">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Magento Layered Cache</title>
<meta http-equiv="robots" content="noindex,noarchive" />
<style type="text/css">
<!--
@charset "utf-8";
/************************************************************
 * Core: Magento
 * @Target: Layered Nav refresh
 * @author: Chris.McKee
 * @date: 23-07-2009
************************************************************/
 
html{ font-size:16px;color:#000; background:#F9F9F9; } /* Set BASE Font Size */
 
/* Reset Browser Differences */
*,body,div,dl,dt,dd,ul,ol,li,h1,h2,h3,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0px;}
table{border-collapse:collapse;border-spacing:0}
address,caption,cite,code,dfn,em,th,var{font-style:normal;font-weight:400}
ol,ul,li{list-style:none}
caption,th{text-align:left;}
h1,h2,h3,h4,h5,h6{font-size:100%;font-weight:400}
q:before,q:after{content:''}
fieldset,img,abbr,acronym{border:none;font-variant:normal;}
sup{vertical-align:text-top;}
sub{vertical-align:text-bottom;}
input,textarea,select{font-family:inherit;font-size:inherit;font-weight:inherit;}
input,textarea,select{*font-size:100%;}
legend{color:#000;}
html, body { height:100%;	width:100%;	}
/*EORESET*/
/* Now thats over we can begin */
body {font:0.8em "Trebuchet MS", Helvetica, Verdana, Arial, FreeSans, sans-serif;}
h1{
	font-size:1.6em;
	position:absolute;
	left:4px;
	display:block;
	top: 8px;
}	
#refresh{
  position:absolute; 
	left:-1000px;
	height:0px;
	width:0px;
}
#counter{ position:relative; margin: 25% auto 0px; border:0px; outline:0px;}
#counter * {font-size:5em; text-align:center; }
small a{ position:absolute; bottom:8px; right:8px; display:block; text-decoration:none; color:#666; }
 
-->
</style>
<script type="text/javascript">
//<![CDATA[
//40 Seconds (no point in changin this)
var intervalID;
function startREFRESH(){  intervalID = setInterval(reloadREFRESH, 40000); }
function reloadREFRESH(){ refresh.location.reload(1); }
function stopREFRESH(){ clearInterval(intervalID);}
//]]>
</script>
<link rel="Shortcut Icon" type="image/x-icon" href="http://chrismckee.co.uk/favicon.ico" />
</head>
 
<body onload="startREFRESH();">
<h1>Layered Navigation Refresh From Browser</h1>
<iframe src="badger-count.php" id="counter" name="counter">You At Least Need a Browser that Supports IFRAMES</iframe>
<iframe src="badger-cm_LayeredCache.php" id="refresh" name="refresh" >You At Least Need a Browser that Supports IFRAMES</iframe>
<small><a href="http://chrismckee.co.uk/?s=layered+navigation">ChrisMckee Designs</a></small>
</body>
</html>

/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/

  • How many records to process at once: Dont Bother changing this
  • StoreID: I’ll let you guess, but for most of us mortals 1 is the default.
  • HOWTORUN: As with the CLI version you can ignore this

To run simply upload to the root of the magento site (as with all the other instructions) and open the badger-wrapper in your browser :o)

Let me know how you get on and and issues

Previous Versions

DOWNLOADS (updated 20090728)

TROUBLESHOOTING *BANG*

Many people on shared hosting experience the following error

Failed loading /usr/local/Zend/lib/ZendExtensionManager_TS.so: /usr/local/Zend/lib/ZendExtensionManager_TS.so: cannot open shared object file: No such file or directory

In this case it’s apparent that your CHRooted/shared environment isn’t set to allow direct access to Zend as required by Magento; in this case its often a good idea to contact your Web Host to see if this is something they can resolve for you.

To show its not this script that causes the issue try the following standard import of Mage…

<?php
//Create a file called testmage.php and add the following code and run. php -qf testmage.php via SSH
//Errors don't always appear in shared environments when using a browser.
require_once("app/Mage.php");
?>

The only “solution” is to run the browser version which acts more like Magento does, though with this mileage may vary

Thanks to Oleksiy for letting me into his server to diagnose this issue.

 

#2 The indexing finishes but my layered navigation doesn’t appear: I’ve heard this one 5 times now and happily its always the same problem… It’s not appeared because you’ve not turned it on.

Required attribute conditions to be shown on layered navigation filters:

  • Attribute properties / Catalog Input Type for Store Owner = Dropdown or Multiple Select (or the System Attribute, Price)
  • Frontend properties / Use In Layered Navigation = Filtered *
  • System properties / Data Type for Saving in Database = Integer
  • System properties / Globally Editable = Yes
  • Please note that a category should have the ‘Is Anchor’ property set to Yes in order for the layered navigation block to show.

Source: http://www.magentocommerce.com/wiki/general/layered-navigation

166 thoughts on “Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck

  1. <del datetime="2009-07-24T09:38:44+00:00">Hey Chris,

    I was fixing to try this out. Looks like this will not get the correct product ids if not starting from 1 and incrementing by one.

    I modified this code to grab the product ids as an array.

    you can also count the collection size by using "getSize". for what its worth

    getCollection()->getAllIds();Mage::getSingleton('catalogindex/indexer')->plainReindex($productIds[$i], null, $sx);</del><ins datetime="2009-07-24T09:38:44+00:00">Problem resolved (CMCKEE – 24-07-09); Thanks for the feedback</ins>

  2. Hi Chris, many THX for update! :)

    If you will find any of the free hosting that can be used for Magento please let me know. Thanks

  3. Ssh ver. isn't working for me, i got:

    <pre lang="php">$ time php -q -f cm_cache.php

    PHP Warning: Unknown(): Unable to load dynamic library '/usr/local/lib/php/20020429/mssql.so' – /usr/local/lib/php/20020429/mssql.so: Undefined symbol "OnUpdateLong" in Unknown on line 0

    Parse error: syntax error, unexpected '{' in /www/avrorus/www/htdocs/products/cm_cache.php on line 12

    real 0m0.220s

    user 0m0.008s

    sys 0m0.005s</pre> </code>

    </pre>

  4. I'll try now browser version, it's run very good, but i don't see counter of how much records done, just header and small link to you site at bottom. Only thing i change was howtorun 2 in badger-cm_LayeredCache.php, i shouldn't do it?

  5. PHP CLI VERSION TEST –>

    Downloaded direct from the site via wget to my test server…

    <pre lang="vi">

    -bash-3.00$ time php -q -f PHPCLI-LayeredCache-vbadger.php

    1136

    start loop (max = 801) (count = 1)

    #Record 1#CURRENT MEMRY USAGE = 6723868;

    #Record 2#CURRENT MEMRY USAGE = 9215184;

    #Record 3#CURRENT MEMRY USAGE = 9405564;

    #Record 4#CURRENT MEMRY USAGE = 9608044;

    #Record 5#CURRENT MEMRY USAGE = 9813064;

    #Record 6#CURRENT MEMRY USAGE = 10042960;

    #Record 7#CURRENT MEMRY USAGE = 10248712;

    {PRESSED CTRL-C TO TERMINATE AS SCRIPT RUNNING FINE}

    </pre>

  6. i download browser ver, change line $HOWTORUN = 1; to $HOWTORUN = 2 and after start get this error:

    5390

    <pre lang="vi">

    ERROR: exception 'Exception' with message 'Warning: fopen(cm_cnt) [function.fopen]: failed to open stream: Permission denied in /var/www/vhosts/apolon.lt/httpdocs/badger-cm_LayeredCache.php on line 77' in /var/www/vhosts/***/httpdocs/app/code/core/Mage/Core/functions.php:239 Stack trace: #0 [internal function]: mageCoreErrorHandler(2, 'fopen(cm_cnt) […', '/var/www/vhosts…', 77, Array) #1 /var/www/vhosts/***/httpdocs/badger-cm_LayeredCache.php(77): fopen('cm_cnt', 'w') #2 {main}

    </pre>

  7. @S: change HOWTORUN back to 1, if you read the article first you'll note that it says method 2 is for a later development.

    If you still get the same error try setting (or creating two empty files of these names and setting) cm_cnt and cm_pd to CHMOD 777; as your server may restrict permissions on writing files via php.

  8. i creating cm_cnt and cm_pd, now no error, but now get this in screen and nothing happening:

    5390 start loop (max = 101) (count = 1)

    after again run get this:

    5391 start loop (max = 202) (count = 102)

  9. @S: The script will continue to run in the background; from what I can see with the error message you showed previously your running the layered cache file direct rather than the wrapper. As per the instructions open the wrapper in your browser. Not the codebehind. Let me know how you get on, my emails on the contact page if you get stuck.

  10. Now everything fine, complete. If i want use cron job run with wget or lynx, i need run PHPCLI Layered Cache or browser Layered Cache ?

    Thanks for nice script!

  11. If you want to use a cron job you can use the PHPCLI version.

    Daniel Craig (wearebeef.co.uk) used this…

    running the following task:

    <pre lang="vi">

    0 */3 * * * root nice -n 5 php -qf /var/www/reindex_nav.sh

    </pre>

    reindex_nav.sh

    <pre lang="vi">

    #!/bin/bash

    nice -n 5 php -f ./update_layered_nav.php

    while [ -e ./cnt ]

    do

    nice -n 5 php -f ./update_layered_nav.php

    done

    exit 0

    </pre>

  12. <del datetime="2009-07-28T18:42:32+00:00">Thanks for the solution! However it doesn't work for me. I've used the browser version. It processed about 8K records but I still don't see layered navigation. And both catalogindex_eav and catalogindex_minimal_price tables are still empty.</del> <ins datetime="2009-07-28T18:42:32+00:00">Fixed by update 20090728 Browser Cache Refresh; PHPCLI edition wont work on Hostmonster.com due to restrictions</ins>

  13. Thanks for help, Chris! How did you script count total products? I see more products in admin, than script is count.

  14. @ogonkov: A lot of the issues regarding running PHP direct from Bash is down to server setups varying (massivley) and various security restrictions and some just dont have zend installed which causes issues.

    The product counts as follows

    <pre lang="php">

    $tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();

    $var = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.

    </pre>

    Which basically says get all ID's from catalog/product collection; we then calculate the size of the array (-1 to account for the array starting from zero) and use that number to grab the record value for the last item in the array (aka the last product) We have to grab the highest product number rather than the number of products because the simple re-index processes based on product numbers.

    For instance if I had 300 products; I deleted 100 out of that and added another hundred, there would still be 300 products but the ID's would of reached 400.

    So if we then processed incrementally though to 300 we would actually miss a hundred records; thus if we grab the highest ID and use that as our processing target, yes we cover some blanks (which takes no real time), but we get to our actual target and therefore process all the files.

    Congrats on getting it up and running :o)

  15. catalogindex_eav table are empty :o

    Store ID seems right.

    I should fix catalogindex_eav table? What i need for this?

  16. Latest update resolves most peoples problems: Clearing Cache Flag was failing due to the new script counting from 1 rather then 0. Please Re-download.

    I'd love to see how many products people are processing with this; which version; and how long (ish) it takes

    Thanks

  17. Part Badger, Part Duck, All Delicious!

    Browser version worked flawlessly with v.1.3.2.2.

    There's still one thing I'm having trouble with: every time I import products via Dataflow, the layered nav is completely killed and I have to run the Badger just to get it back up. Is that meant to be the case? I thought unchecking the 'Layered Nav Cache Control Beta' box in Admin was meant to prevent this.

    Anyone know how to update via Dataflow without killing Layered Nav?

    Thanks again Chris for the great script!

  18. @Will: Good stuff, If you add products via magento admin the layered nav 'should' be updated. Using the API to my recollections doesn't bother and you'll need to run the badger again.

    You can of course cheat if you know where the records start. Simply take the product ID of the first new product added (minus one) and drop that in the cm_cnt file and the script will run from that point in.

    Chris

  19. Hi Chris,

    Thanks for the hard work, I'm currently trying your shh version out. I just wonder, how do you know to which amount you can change the $HM variable? What happens if it's to high? I just altered it at my script to 10000 (however we have currently only 8000 products in store), but the script does seem to run fine.

  20. @E: If the number is too high the script will eventually run out of memory and stop. When it stops just restart it.

    It will cause absolutely no real issues as the persistent count and limit are stored in cm_cnt and cm_pd till the script count = total anyway. The number I put it was just an arbitrator number plucked out of the air that reached close to the max my shared host test setup would handle. (128meg Max for a php script).

    If you've got the memory you could run it in batches of 1000; I cant see it completing in one run without running out of memory trying to run 10000. But as above, it wont cause an issue.

  21. Hi, i try like you write to start from cron job

    1. write in cron: <pre lang="vi"> nice -n 5 php -qf var/www/vhosts/httpdocs/reindex_nav.sh</pre>

    2. create new file reindex_nav.sh with:

    <pre lang="vi">

    #!/bin/bash

    nice -n 5 php -f ./update_layered_nav.php

    while [ -e ./cnt ]

    do

    nice -n 5 php -f ./update_layered_nav.php

    done

    exit 0

    </pre>

    3. rename cm_LayeredCache_badger.php to update_layered_nav.php

    and after run get cron email just with this:

    <pre lang="vi">

    nice -n 5 php -f ./update_layered_nav.php

    while [ -e ./cnt ]

    do

    nice -n 5 php -f ./update_layered_nav.php

    done

    exit 0

    </pre>

    i dont know we the problem, maybe can help?

    after i try in cron write like same for ssh:

    <pre lang="vi">

    time php -q -f /var/www/vhosts/httpdocs/cm_LayeredCache_badger.php

    </pre>

    and get cron mail this:

    <pre lang="vi">

    ERROR: exception 'Exception' with message 'Warning: fopen(cm_pd): failed to open stream: Permission denied in /var/www/vhosts/httpdocs/cm_LayeredCache_badger.php on line 40' in /var/www/vhosts/httpdocs/app/code/core/Mage/Core/functions.php:239

    Stack trace:

    #0 [internal function]: mageCoreErrorHandler(2, 'fopen(cm_pd): f…', '/var/www/vhosts…', 40, Array)

    #1 /var/www/vhosts/httpdocs/cm_LayeredCache_badger.php(40): fopen('cm_pd', 'w')

    #2 {main}

    real 0m0.195s

    user 0m0.106s

    sys 0m0.069s

    </pre>

  22. @S: It looks like a permissions issue, Warning: fopen(cm_pd): failed to open stream: Permission denied usually means that the user CRON or Apache is using isn't the same or in the group required to run that script and access that file.

    You could always try SU cron; then running the function that way to see if it has the same issue.

    Unfortunately one of the joys of the linux os :o)

  23. I try start from ssh and work perfect, i try with default client login(same like cron) and try with root, when i try with root i found cm_pd and cm_cnt in /root/ folder, i don't thing its correct, maybe need in script write full where file locate cm_pd and cm_cnd like /var/www/vhosts/httpdocs/cm_pd i dont know php, how i can do this?

  24. i change $prodCount = "cm_pd" and $countFile = "cm_cnt" to

    $countFile = "/var/www/vhosts/httpdocs/cm_cnt"

    $prodCount = "/var/www/vhosts/httpdocs/cm_pd"

    and now file create without problem and error. What about $HM = 800; when run from ssh her need write max value how many product i have? I have 4500product, when her write default 500 i see in ssh like max execute 500. If i write 10000 script execute all my product.

  25. You can change the max value to what ever you like; but if you run out of memory the script will terminate; restarting the script will then continue from where it left off.

    If your using this script theres a good chance that you cant run processes for long enough to achieve running the entire catalogue in one go; otherwise the one built into magento would do the job.

    I'd usually advise running the script manually starting at a high number (say 700 for 128 ram) and seeing if it falls over; if it dies lower the number if it runs successfully you can try increasing it.

    Chris

  26. Chris, thanks so much for your work on this and for making it available. I did my best to read through all the comments on all three posts, and I have a basic question that I couldn't find an answer to anywhere else.

    We're running Magento 1.2.1, and started having problems with the layered navigation. Under 'Cache Control (beta)' we had the 'Layered Navigation' box checked, and every day, the layered navigation breaks. When I go into the cache control section of the admin, next to the "Layered Navigation Indices" there's a button that says the indexing is running, but of course it never finishes. To fix it, we've been clicking the button next to it "refresh now" and that asks if we want to kill the current process and start a new one. We do, and then it works until the next day.

    From what I understand from reading your posts and the comments, unchecking the "Layered Navigation" under "Cache Control (beta)" should stop that daily refresh. Is that correct? Is that all that setting does? Or more clearly, if I turn off that setting, but go in daily and manually hit the 'Refresh now', will that do the same thing?

    Is your script intended to fix the 'auto refresh' of the indices, or is it for those for whom hitting "Refresh Now" doesn't even work?

    Thanks for your help!

  27. @Luke Berg The script is a TOTAL replacement for the admin layered navigation.

    If you Got to the ‘Cache Control (beta)’ and turn off the layered navigation then run my script. The first thing the script will do is clear the flags that say the process is still running.

    The problem is the amount of time to process records vs the max-execution time for the script and the amount of memory leaked.

    Soooo if your on a 4GIG ECC Ram, Multicore, dedicated box you can probably get away with running the built in script with a few php.ini edits.

    Otherwise disable it and use mine :o)

  28. Sorry, I don't think that answered my question, or if it did, I misunderstood it. :-

    Unchecking the box "Layered Navigation" under "Cache Control (beta)" does not turn off layered navigation, because I still see it working on my store, and I can refresh it manually by clicking 'refresh now' next to 'layered navigation indices'. What I'm guessing it does (but can't seem to confirm anywhere) is automatically do exactly what clicking 'refresh now' does.. just updates the layered navigation indices.

    For us, the automatic daily procedure fails, but I'm able to do it manually.

    My questions are, does that check box do more than I described above, that you know of?

    Is your script intended to be run automatically on a schedule or manually?

    Thanks!

  29. It 'attempts' to run exactly the same process yes.

    The 'browser' version of this script is more for people on shared hosting (though anyone can use it)

    The PHP-CLI Version can be run on a cronjob.

  30. Chris,

    I downloaded Browser Layered Cache – vbadger – v1.3.4 and uploaded and ran it. It ran through 4100 files and finished with a screen saying Files Deleted on the url badger-count.php?done=1. I read through this numerous times to see if I missed something, but I think alot of it is over my head. My admin navigation does not expand when I mouse over. Is this what your program is designed to repair? I really need help. I have been dead in the water for about a week now.

    Thanks, Bill

  31. @Bill Nash: The script is a fix for the Cache Management – Beta Cache – Layered Navigation. As detailed in this video http://www.magentocommerce.com/media/screencasts/… .

    The files it deletes is just the two temporary files (counter and total).

    This won't fix your Admin Navigation error; Try disabling your plugins first to see if they're causing your admin-nav issues, and theres always the possibility of reinstalling the admin/admin theme via magento connect.

  32. We've been having a ton of trouble with this. I can set the $HM to 100,000 to do the 81k products in our store in one pass. It completes, but most of the time we get nothing showing but price, sometimes we'll get manufacturer, sometimes nothing at all. Very frustrating.

    Also, is there a way to do multiple stores at once? I can run it for each of the 7, but that takes a long, long time. It's over 2 hours per store as is on a dual dual-core box with 8 gigs of RAM for both web and db servers. We have 4 more web servers besides this one with similar setups, other than the one with 32 gigs of RAM that we also use for memcached.

    Anyway, what happens if I just remove that $STOREID variable? Will it do all of them at once? I've got a few hundred thousand products to add yet, so I'm sure I'll need to do some tweaking to get this working when we're all done.

  33. @BrianMark if you remove the store id from the main function in the loop, the singleton would in theory run through the product id on all stores. Give it ago and let me know.

    Oh and JESUS THATS A NICE SETUP :op <insert – server envy>

  34. I found a error in the line number 99 in the CLI version.

    Original:

    <code><pre lang="php"> getStore('1');</pre></code>

    Fix:

    <code><pre lang="php">getStore( $STOREID );</pre></code>

    Thanks for your script! :)

  35. @Gabriel Thanks I'll changed that in a mo or so. Testing code left in me thinks!

    Fixed in PHPCLI + Browser edition, issue will only affect people with multiple stores. (or stores whose ID is not 1).

    Teach me to write entire scripts in SSH/VI :op

  36. BTW, removing the store id just causes an error. I might have to create a loop so it'll do all 7 stores (and counting, right?) in one pass.

    It'd be nice if the script could only do the products in a store instead of all, though. We've got one store that will have 600,000 products, one that has 100, one with 2,500, and one with 15,000. Doing all 600k+ for each store is very inefficient. Any thoughts there?

    Thanks for the great script.

  37. @BrianMark where I count the array size of Products to get the max-ID you could 'technically' go on to use that array to process just the records that are there. The only downside to this is you lose the ability for the script to fall over or be stopped and restarted from its previous place.

    Really to do multiple in one run you need to add a new count file (for store id) and loop the whole script; that way the script will run the 600K for the first store, wipe the count files and move to running just the 100 for the second and so on. Shouldn't be too hard to write really but I don't have test-bases with multiple stores to whack out the changes I'm afraid.

  38. Chris,

    I have the wrapper version of Badger 1.3.5 installed on a test site running Magento 1.3.2.2 on a dedicated Lightspeed server with dual CPUs and 32GB of RAM. My store has about 62,000 grouped and simple products with 109 Magento attributes that have been imported through a customized Magento import profile. The issue is Badger runs successfully (no errors), creates cm_cnt and cm_pd files and deletes the files successfully after I click the link but layered navigation does not change from before I ran the script. I have run the script several times with no results and no errors. I'm guessing there is one instruction you mentioned somewhere that I am missing.

    Note: After the script runs, both files only contain the text "108988" before I delete them. This does match the count I saw on the html page as it was counting down. I appreciate the help. – Fred

  39. Hi Chris. Just noticed this update. Runs fine on my server, but on another server I'm helping with whilst trying to run the web version I get this error –

    A file permissions error has occurred. Please check the permissions on the script and the directory it is in and try again.

    All files are set to 777. Any ideas? Thanks

  40. @Fred: Hard to say really; try looking in your Cache management options in Magento Admin and see if it still says that the layered cache is still running. The script should clear this flag automatically so it shouldn't, but if it hasnt the index wont update.

    @James: The PHP scripts should be set to 755; if you monitor the folder you should see the cnt pd file created. If it isnt you may need to pre-create these two files (empty files) and set them to 777. Some security settings are known to cause problems with file creation.

  41. hi Chris, firstly thanks for all you have done to the Magento community.

    Now I wanted to let you know that the script ver that I have donwloaded from this site: http://jksite.com/blog/2009/07/magento-layered-na… didn’t work for me, I don’t know why!

    It is connecting to the DB but once the layout comes out it says this, try it by yourself: http://nationaltree.com/mag_store/cache.php
    and nothing happens, nothing changes!

    Any idea why?

    I also would like to know how to download this 3rd version in a zip file, if possible!

    Thanks again!

  42. @Segemel: Not too sure as the one that JK branched from was the Part Deux script and uses none-mage-api connections to the database.

    The second thing I can help you with though, the PHPCLI version zipped :)

    Zipped PHP File

    This version is to be run from SHELL or via SSH.

    If you dont have access to that use the Badger-wrapper which is already zipped.

  43. Thanks budy!!

    It made my Size sort list to be shown, now just 1 more thing

    Take a look to my website on Christmas Trees Tab, you'll see that my Size and Price sortting list is not using all my product comparing it to my category sort list.

    Do you have an idea why is that happening, weird ha?

    thanks in advance!!

  44. Chris,

    Thanks for the feedback. It does appear that the script is unable to stop the Magento refresh process once it is started. Even though I had unchecked the beta layered nav. refresh box before uploading products, once they were imported, Magento started the refresh, I started Badger which was unable to stop Magento from running the refresh process. The count on the script kept bouncing from approx. 124,000 to 102,000 back up to 110,000 but never got below 102,000. I'm guessing this was because it couldn't stop the Magento refresh.

    Should the PHPCLI version potentially behave any different? Any other suggestions? Thank you.

  45. chris,

    thanks for your great script, i've used it successfully on my shop via browser (i got the "files deleted" screen). however i still can't get the prices on grouped objects in layered navigation to show! i'm banging my head to no avail. do you have any suggestions?

  46. Hi Chris, thanks for this great script. I am using the browser version of your script and am on Magento 1.3.2.1. I clear the cache and any flags in the core_flag table. Everything is turned off in the cache are on the backend of Magento. I then run the wrapper and everything seems to go perfectly in the browser. However after multiple attempts and checking I see that as soon as the wrapper is started a catalogindex flag pops up in the core_flag table. So although the script goes through all the products when I look at the layered nav on my site many of my custom attributes are incorrect by a wide margin. Price, category are correct but my custom attributes have only 0-50 products showing when it should be closer to 3500. Any ideas on why this (catalogindex popping up in core_flag as soon as script is started) would be happening and what I can do to fix it? Does having flat category or flat product enabled have any negative effects on the script? Thanks for your help.

  47. @jonesy not quite sure there's the following main singleton methods related to cache;

    * Mage::getSingleton('catalogindex/indexer')->plainReindex(); – Layered Nav

    * Mage::getSingleton('catalogsearch/fulltext')->rebuildIndex(); – Search Index

    * Mage::getSingleton('cataloginventory/stock_status')->rebuild(); – Stock Cache

    * Mage::getSingleton('catalog/url')->refreshRewrites(); – Catalogue (URL) Cache

    Theres probably a few more since the inception of new beta-cache methods in the more recent versions so I'll have to check.

    @Brian Mark: Not sure what you mean; email me and send a screen shot my heads full of all sorts at the moment.

    @len3121 The main problem that causes the flag to reappear (as noted by Fred) was Magento itself; If a product is updated / imported / added then the layered navigation will be kicked off again.

    Cant really say anything about the flat-category/product options yet but that could be causing an issue; try regenerating the flat files post-index. I'll have to look into this later so if this method works for you let me know and I'll look into combining the functionality.

    Chris

  48. Ok, so it had to do with making our first level categories no longer anchors. If something matched there, we only showed 1 under the main category instead of how many were in categories under that. Made it pretty useless. All better now, although I don't like categories with 2000+ matches.

  49. Hi. Ran the script and it runs fine. I have a store that is used for development

    real 2m1.434s

    user 0m32.788s

    sys 0m0.637s

    Is the script designed to solve the following problem.

    1. Navigate to From admin -> Catalog -> Manage Categories,

    2. The categories show the wrong number of products in the categories!

    3 For example, a category 'Pets' which has only 4 products, yet the category management page lists 130.

    Has anyone encountered a similar issue? Or know how to go about troubleshooting?

  50. @daniel yours sounds like a glitch in counting on the admin-catergory management which is an odd one; have you tried a) clearing all cache and b) adding/removing a product to see if it affects the count.

    Chris

  51. Thanks for the script, I am using the Browser Edition and I have a problem with the ID count, I have my highest ID is 19,833 and the script keeps counting only 16,710. I added about 100 more products and script keeps giving the same 16,710 count.

    Do you have an idea what the problem can be?

    Thank You

  52. The files are been deleted properly after each script termination. What I found looking at my product listing was the product # 16709 was the only one I had with multiple select attributes, so I deleted it to see if that was the problem but the ID count went even lower to 16371. I am looking at the products around that ID but there is nothing unusual.

  53. @N Lopez: The "count" is the final product ID in the product-array. If you believe its wrong you can manually create cm_pd (the total) prior to starting the script, literally just stick in the number you think it is.

  54. I started to use the browser version, and it started to work and went through all the 6000 products but the indices only updated through like the first 2000k products. I have "Layered Navigation" unchecked. The one thing I did notice though is that "Layered Navigation Indices" still has "running… kill" next to it.

  55. as far as I know magento rebuilds Layered Navigation for all products after import, not only for that were imported. How do you think may be it will be better to rebuild only for that were imported?

  56. Hey its me again. I gave this page to my host (I told him the browser version was only working for half of my products) and he did the CLI version of this test and it worked perfectly. My question is I've read this article (and the previous two) and I can't figure out how to correctly run it. I've never setup a cron job before. Now don't get me wrong, I know CSS, HTML and all that good stuff, but I would say I am new to database management. Could someone explain the proper steps to do the CLI version to me? Thanks

  57. @Anton all automatic indexing is disable-able; but ye its a pain in the arse, if you stop all the automatic indexing you can use the script on its own. Of course I'm going to disclaim here that I don't advise the manual editing (rather than over-riding) of Mage Core scripts.

    @Kris Whos your host?

    As for the CRON job your host should be able to give you instructions but here's a generated script to for a daily run…
    http://www.webmaster-toolkit.com/cron-generator.s

  58. Hi,

    Is it supposed to run for a very long time? It's been refreshing every 5 sec. for about 15 minutes now;-)

    Hopefully this is correct and it will end a year of troubling layered nav ;-)

  59. @Paul yep time depends on the number of products + attributes; Every 5 seconds seems slow so you either have a lot of attributes or a slow Database Setup; Normally it takes a second or less per product.

  60. @Paul: Have the "count" files been created? If not you may need to do it manually; accessing badger-count.php id expect to see the output of the current progress (less you've deleted them already); if not does your site have error-output turned on?

    Try running the CLI version from SSH…

  61. I have 2 cnt files. 1 cnt, 1 cm_cnt. The first one is possibly from your old version;-)

    The cm_cnt is created today and contains the number 1300-something, which would be about right.

    I'd love to have the browser version working instead of messing around with Putty ;-)

    Anything we might have missed?

    ————-

    Ok, i copied cn_cnt and renamed it cnt_pd, it worked fine i guess.

    [Count: 1317 | Total: 1317][ 0 to go]

    Click Here To Complete and Delete Count files

    But… maybe i don't get it, i still don't get any of the layered nav options other than Category: http://www.espressoshop.nl/thee.html

    *slightly embarrased

  62. @Paul: cm_cnt Should Start at Zero; cm_pd should be automatically created with the total of your products in it (i.e the highest ID)

    Copying CM_CNT -> CM_PD would essentially say the process is complete.

  63. Screw me, it works now as it should. This made me think:

    "change HOWTORUN back to 1, if you read the article first you’ll note that it says method 2 is for a later development."

    * more embarrased;-)

  64. Thank you for the prompt help. Before posting i was already amazed by your involvement in the product and comments. My posts today prove nothing less than just that: commitment and pleasure in your own stuff.

    Two thumbs up, keep up the good work!

  65. Chris, still no dice but i may have found why. You might have a final word on this:

    I exported all products and noticed this value for each item, regardsless the store they are mainly for: store_id = 0.

    In your script this is set to 1. Now i do not know if this is an important setting, but i have set it to 0 and it is runnnig now. You think that might be it? (411 to go… ;-))

  66. @Paul I cant reproduce the problem in any test bases; but The storeid should match the store id you can see in the URL string in the Magento Admin (store management). Generally if you've setup a single store that ID is one.

  67. Ok, we have it up to 15 up there, however i have been testing store 1 all the time. Will tryo some others too.

    Still nothing. Too bad we have a lot of great attributes lined up for searches :/

    1. Hi,

      sry to bother you with an old magento version, but i have exact the problem that paul has. Lots of stores, and changing storeid doesnt matter.

  68. Chris, you asked for stats, so here we go:

    #Record 101#CURRENT MEMRY USAGE = 10154748;

    real 13m39.059s

    user 6m25.255s

    sys 0m14.638s

    As you can see it's much slower than your results. We're on Magento 1.3.2.4 with 15.600+ products.

    Thank you for all your effort on this issue. Can't wait for this to be fixed, hopefully in 1.4

  69. @Rogier Bom: I found that running the new code-compiler made all of this go a lot faster, of course alot depends on your setup, memory, db etc. Big queries take along time.

    The script should still be useful for timeouts (which will still occur) the optimizations on 1.4 simply condense the query and make it a tad less painful for the CPU/DB/MEMORY.

    Chris

  70. Do you have any experience with the new compiler in a production environment? I'm a bit hesitant, but with the number of products in this store things are starting to slowdown bigtime. Let's see what it does in the test environment first.

  71. @Rogier Bom: I actually ran it on a staging server first then uploaded the whole site; not had any problems yet *touch wood* but I have had a decent performance improvement.

    It "MAY" improve indexing speed but take into account the majority of the time delay on the indexer at the moment is the time between the transaction statement being created in the PHP and how long it takes to execute per product.

    Optimizing your database setup to match their enterprise whitepaper is going to achieve more in the long run.

  72. Thanks Chris for this solution. I have a dumb question tough. After you download the files where do you copy them? I know that there's a folder called magento root but I don't know the location of that folder. I have vps hosting so I have ssh access too. Please include as much details as possible as I'm not very familiar with linux system.

    Thanks in advance

  73. I seem to be having some troubles with setting up the cron. The instructions you gave are perfect, but it doesn't seem to loop through the count. I have 28000 products, and occasionally it will give an our of memory error goìng through 1000 products at a time. Is there a way to get the reindex_nav.sh to loop through till its complete?

    Thanks in advance!

  74. hi,

    THANKS so much for spending yore time to make this script and for sharing it free of charge! i just used it to fix my LN problem and the browser version worked perfectly!

    (p.s im on shared hosting if that helps!)

    how do i cancel magento trying to run the indexing (so i just do it manually with the cheeky badger) and it doesnt mess up my whole nav system?

    many thanks!

    chris – manchester uk

  75. @Jason hasani: just use FTP; there should be a folder called public_html / WWW or something like that (depending on the setup) which will contain the .htaccess / index.php, app folder; basically all of your magento setup; thats the root of magento.

    @Troy: if you can work out how long it takes to do a "run" at a limit that doesnt burst your memory GC then you can create an extra shell script like below

    <pre>

    >> FILE NAME reindex_nav.sh

    #!/bin/bash

    nice -n 5 php -f ./update_layered_nav.php

    while [ -e ./cm_cnt]

    do

    nice -n 5 php -f ./update_layered_nav.php

    done

    exit 0

    </pre>

    As I posted (http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-badger-part-duck/#comment-183)

    Basically that should run the php file till the cnt file is destroyed :)

    Obviously modify the script to use the file names you have (i.e. update_layered_nav.php > cm_LayeredCache_badger.php)

    @chris wood: Hello from Bolton; Basically you need to turn it off in the nav… then if magento is still being a git you can nip into the indexer file and disable more bits of it. I try to avoid this as it means editing CORE files; but if its a regularly updated site then you'll find that magento tries to reindex after you add products, edit products change category details etc etc etc.

    Good news is; Magento 1.4 has been optimized quite alot; and post wedding I'll be reviewing the changes in a long winded follow up to my request for data.

  76. Hey chris mckee,

    thanks for the feedback. is the badger-wrapper only supposed to run once? i navigated to it a second time but it doesn't really do much..in fact it does nothing just displays this message "Layered Navigation Refresh From Browser"..

    i really liked seeing it counting last time, was nice to see the progress as opposed to with the standard magento system!

    thanks

    chris

  77. @chris wood: Basically you run it and it runs 2 scripts in iframes; One that does the processing and the counter file which just shows you your progress based on the the count and final count files; the counter script also handles the bit at the end where it offers you the chance to delete the two count files and end the script.

    If you had issues check to make sure that the two count files were deleted at the end; as on some setups PHP doesn't have file system delete permissions.

  78. Hi Chris McKee, thanks for that…sorry if i'm being really dumb here but where would i find the count files that should be deleted? i think that this should be the problem as i cant think of anything else… thanks in advance chris

  79. Hi Chris McKee, ignore my last post please, I actually found out that they should be in the same folder (by running the script on a different magento installation) but they are not there so they were actually deleted when i ran the script the 1st time. I copied fresh badger files into my original directory and it still refuses to run again the page loads but it seems like it fails to initiate, it just keeps refreshing every 10 seconds or so. so right now it is running fine on my other install for the first time using the same files…..(this is a live post, im almost commentating as it goes!) ok so, the file ran fine on the new directory for the 1st time, and just ran it a second time and that is fine too! confused. so now i need to work out why it isn't working on the original directory but it is on the second directory – how frustrating that it works in my test enviro but not my live enviro! these are things i am thinking…could it be: 1. layered nav indexing is running in the back end of my magento install? i tried to stop it but it just queues ?? 2. that the directory is password protected using apache – pretty sure that this doesn't make any difference… please help if you can, not having my layered navigation is really worrying me! regards chris

  80. @ Brian Mark

    We had the same problem where only price was showing after. It was due to a module called Lucky_InStockOnly that hides out of stock products but the creator added new code to make it work.

    Anyone else who has this module and the same problem look at the updated blog entry. Took me forever to figure out what was going on.

    http://colin.mollenhour.com/2009/09/yet-another-h

  81. Since Chris is busy getting married (congrats btw) I figured I would ask if anyone looking at this thread knows how to stop the browser version from running? i tried running it on a different store id for my subdomain but i must have made a mistake since all I see is [Count: | Total: 3392][ 3392 to go] and that never changes. I assume it is not seeing the count file and will perpetually run. Any way to stop it?

    1. @len3121 The browser version offers an option on completion to delete the two files it creates. If you've closed the window prematurely simply delete the two counter files.

  82. I have used the browser version and it seems to be successful. I am trying to use this to be able to sort my products by price once again. Even after running this, the products won’t sort correctly. Do you think I am doing something wrong with this script or is it possibly another error?

    1. @andrew I think someone asked this one before and it was related to the new compiler. My heads a tad fuzzy at the moment so when it clears up i'll have a prod around.

  83. hey hey,

    I got a problem, my navigation keeps on running.

    I even changed to a complete new server with magento hosting.

    i kept the same problemen, can't run it.

    Now i uploaden you script, everythings seems fine but it keeps saying:

    [Count: | Total: 2731][ 2731 to go]

    Nothing happends? Please help!

  84. Why do I get an error 500- internal server error when trying to run the file?

    On the browser version

    magento 1.32.3

    1. Check the file permissions, and check your apache/lighttpd ~ server error log for details about the error. Turn on PHP error mode, probably wont make a difference though; your logs are the best bet.

  85. Hi there. I LOVE your script. works wonderfully. My question is the mangento Layered Navagation Intices keeps running automatically everyday, fucking up my layered navagation. do you know how to stop it from running on its own?

  86. @Chris – Thanks for replying. Are you referring to the messages above to/from Rogier Bom? I took a look at those but wasn't able to find anything to help me.

  87. Can't access the error logs. Where would I add this script to create an error log and where would I find the file ?

    I have the permissions set to 777

    Creating own error logs for debugging PHP scripts.

    Since we don't provide access to Apache error logs on shared hosting packages for

    technical reasons, you can create your own error logs for debugging PHP Scripts.

    Please insert the following code in your PHP script (or create separate file and

    and add the code in it. Include the file using "include()")

    error_reporting(0);

    $old_error_handler = set_error_handler("userErrorHandler");

    function userErrorHandler ($errno, $errmsg, $filename, $linenum, $vars)

    {

    $time=date("d M Y H:i:s");

    // Get the error type from the error number

    $errortype = array (1 => "Error",

    2 => "Warning",

    4 => "Parsing Error",

    8 => "Notice",

    16 => "Core Error",

    32 => "Core Warning",

    64 => "Compile Error",

    128 => "Compile Warning",

    256 => "User Error",

    512 => "User Warning",

    1024 => "User Notice");

    $errlevel=$errortype[$errno];

    //Write error to log file (CSV format)

    $errfile=fopen("errors.csv","a");

    fputs($errfile,""$time","$filename:

    $linenum","($errlevel) $errmsg"
    ");

    fclose($errfile);

    if($errno!=2 && $errno!=8) {

    //Terminate script if fatal error

    die("A fatal error has occurred. Script execution has been aborted");

    }

    }

  88. @craigcriv file should be set to 0755 (or if your using an ftp client it may only accept 3 numbers, these would be 755)

    That script could be included at the top of each PHP file after the <?php

  89. Thanks Chris Got it to work, had wrong path ( /magento/badger-wrapper.php )

    Should not have added magento in the path.

    I did get the layered navigation to appear but not all the manufacturers are showing up n the search by only one, and the item counts in the price or manufacturer are not correct.

  90. i have 2 questions. 1st how can i run this script from cron? I don't have SSH access though. and 2nd magento's builtin layered navagation indices keeps running automatically. how can i stop it? i've already removed it from the cron scheduler

  91. We spent countless hours trying to get the Refresh of Layered Navigation Indices to work.

    We had close to 10000 products on pacwave.com runnning on a VPS, and nothing we did worked.

    The solution was to add RLimitCPU 1200 to the .htaccess file. This basically prevents the refresh script from consuming too much of the CPU resources and crashing.

    Hope this helps some of you.

  92. I've run this script successfully a few times but occassionally I get the following error. The script completes, I delete the files through the link, refresh the cache. Now I have a working front page, but all of the links from the featured products on the front page and the links in the nav bar give the following:

    Not Found

    The requested URL /home-electrical.html was not found on this server.

    Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

    Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.7a DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 Server at http://www.xxxxxxx.co.uk Port 80

    I've managed to get the site to work again in the past by some weird combinations of rerunning the script and refreshing the cashces but this time it seems to be stuk completely! All of the items (4,900) and categories are still showing in the admin panel! Any ideas?

    Cheers

    Gav

  93. I have a problem with grouped products. I can´t see “Starting at: ….” I solved using “Layered Navigation Indices”. Now I have 2000 products and fails because “my set_time_limit” with my server is 30 seconds. I’m using your script. It runs well (i think…) but do not get the same result as doing it from the admin of magento. I can not see “Starting at: ….” For a long time I am trying to solve this problem, but can not find a solution.

    Can this script help me?

    Thanks a lot.

  94. Hi Chris,

    Your script is great, but it can refresh completely my layered nav.

    I'm using just one attribute for Age in my store for almost 1K products. When I refresh the nav, can't find all available products… approximately 1/3 of them is missing.

    In the past, when I added new products, they appeared immediately in to the layered nav, but since one month it' stopped. The only solution was to use your script, cause the refresh of the Indices never ends. Also, I have a price rule, that stops works in every 2 ~ 3 days and I must recreate it. First I thought, that there is a relation between, but in fact NO, when I stopped the rule, the lay nav stays the same – incomplete.

    Finally, I have an amount of products that is available in the layered navigation, but when I add new products, it's impossible to show them in.

    Cheers form Bulgaria ;)

  95. You are great!

    You gave me an exceptional and professional solution. I run the browser package and boom the scripts did their work.

    Thanks again. I was really stressed with this refresh problem. You helped me solve with the most elegance solution one my pressing problems so far.

    Vpas

  96. The PHP-CLI and browser version work correctly, but I have a question:

    In admin panel the Layered Navigation is disabled,

    however I find that every day starts the rebuild layered navigation index ¿?

    At the time I run your script so that everything goes well.

    What is the reason to launch automatically Layered Navigation Index?

    I programmed in crontab cron.php script. Does this spear?

    Thanks

  97. <code><pre lang="php">

    What is the reason to launch automatically Layered Navigation Index?

    </pre></code>

    I autoresponder, if anyone is interested.

    See appcodecoreMageCatalogIndexetcconfig.xml the default values:

    <code><pre lang="php">

    ………

    ………

    0 2 * * *

    catalogindex/observer::reindexAll

    * * * * *

    catalogindex/observer::runQueuedIndexing

    ……

    ……

    </pre></code>

    Thanks,

  98. Sorry,

    This is the code:

    <cite>

    0 2 * * *

    catalogindex/observer::reindexAll

    * * * * *

    catalogindex/observer::runQueuedIndexing

    </cite>

  99. Hi Chris,

    Do you know what would be the best way to stop magento from automatically trying to refresh the layered nav after a fresh import.

    Basically, I assumed it had something to do with a cronjob.

    As recommended by others, I commented out:

    <!–

    0 2 * * *

    catalogindex/observer::reindexAll

    * * * * *

    catalogindex/observer::runQueuedIndexing

    –>

    in appcodecoreMageCatalogIndexetcconfig.xml

    But still no luck.

    Any ideas, or know where to start?

    Many thanks,

    Eric

  100. Just FYI – my problem turned out to be yet another Magento well known but as yet still unfixed bug – the URL rewrite bug. It had nothing to do with Chris's code at all :)

  101. Do you have a code or app to rebuild the search index?

    I have the same problem with the server timeout like I did with the layered navigation rebuild, which you fixed with the badger wrapper app.

    Thanks

  102. Chris – Thank you for this solution. I have the browser version running on my 9,000 test products and everything appears to be working correctly (script says its completed and I delete the count files). However, when I go to a category that I have set to anchor the layered navigation count does not match the number of products.

    Example:

    Category A has 47 products – I can browse through all of them without issue.

    Layered Navigaton has the following:

    $0.00 – $100.00 (5)

    $100.00 – $200.00 (1)

    $200.00 – $300.00 (1)

    Any reason you konw of why this would be happening? I treid to rebuild the other indexes and that didn't seem to fix it. I have not run a product load since I began testing this script.

    Thanks ahead of time!

  103. Hi Chris,

    a well done script and so necessary as we all need it!

    In my case it does not work correctly. Take a look at yourself: http://www.brady-schweiz.ch/badger-wrapper.php

    I assume this is due to several storeIDs in this site: 0 for admin and 1,2,3 for three languages.

    It starts with 27803 records, but I don't have so many products, only about 16000.

    Could it be, that this count is the total without regard to the storeID?

    I would pay you the hours for adjusting this script to work with several storeIDs.

    These should be gotten from the shop itself or setup by an array.

    Do you have time for this?

    yours

    Res

  104. Hi Chris this script is great, saved me a lot of problems.

    I have one question that I'm hoping you might know the answer to, in our store we have reached now around 2900 products, your script was working fine up till about 2400 but recently when I run it it thinks that it has done all the products but the last 400 or so don't get refreshed. Any ideas how I can fix this?

    Many thanks

    Josh

  105. Hi Chris.

    I got Magento 1.3.1 and I made an upgrade (using Magento Connect) to 1.3.2.4.

    Since that upgrade, layered navigation works but I can't see the price filter.

    I've executed your script, but nothing change on the layered navigation.

    Any idea?

    Cheers.

  106. i got this error :Error while refreshed Layered Navigation Indices. Please try again later

    and i dont know how i can fix it,i use some of ur scripts but i a mlittle confused,where i install this scripts?

    thnaks

    i use magento 1.4.0.1

  107. so i make it right but no one scripts run for my 1.4.0.1,i already contact with my ISP and i google it my problems but nothing,i spend a week and many many hours to find a solution but i lost my time ithink.

  108. Unfortunately the problem this script was made to solve was caused by the layered navigation index script built into magento eating up all of the system time and memory.

    I' cant validate if your having the same issue; have you checked your system resources while its running (need ssh) or does it die while running.

    If you dont have full ssh you can usually get a relatively good idea using http://www.wormly.com/ as its a PHP script. Handy for monitoring memory spikes.

    I've stuck a call-out onto twitter, if anyones still having issues regarding layered indices using too much mem/timing out let me know and I'll have a poke. Otherwise I'll be assuming in the meantime its resolved as of Magento 1.4.1.

    Thanks

  109. heey chris, my problem is the sort by price functionality is acting weird. if the category has products only 1 attribute set, it works fine, but the category has products which have different attributes sets it works like grouping the attribute sets and sorting between them,

    thanks in advance

  110. Hi,

    thanks for this great script.

    One question: We have more than one Store (end storeid)

    If I use

    Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, null)

    (last param null instead of $sx

    will I create LN for all store?

    Thanks.

  111. Hi Chris,

    I tried the web version (wrapper), but i just have the header "Layered Navigation Refresh From Browser" and nothing else. What am i doing wrong?

  112. @walter the default option for indexing is if store is null cycle through stores.

    @mclean thanks

    @nghia turn on logging on PHP and check your logs, something might not be running.

  113. Hi Chris,

    I've been doing some testing with 1.4 . I've imported my 18,000 products, and you're correct it looks like it's able to handle reindexing product attributes, but it always times out when I try to rebuild catalog product fulltext search index.

    Just for reference anyway, it may be something you might to look at in future. Looking at the table, it looks like it gets through about 5500 records before it stops.

    Thanks

    James

  114. Hi Chris,

    Now I'am with version 1.4.1 and I got the same error than James:

    The Catalog Search Index is always at Processing. When I force a refresh, I got a timeout error from my webhost. It's the same kind of problem than before, but not on layered navigation but on fulltext search index…

    Can you help?

    Thanks

  115. Hi,

    thank you for this script !

    I have another question:

    How to delete the cache on my other Store Views?

    English: Store ID1 (works)

    Spanish: Store ID 2 (not)

    France: Strore ID 3 (not)

    Do I need to change something in the script?

    If so, what exactly?

    Thank you!

  116. MagicCardz, James: This script system is written purely for Versions below 1.4; the changes they made to the system (they pretty much rewrote the indexing), breaks this script like a twig. I'll inevitably end up rebuilding it as some people still have issues on 1.4.1 with various bits of indexing. The bad news is magento couldnt give a rats arse about people hosting on servers with less than 256 mb of memory available for PHP, which is plain fucking stupid.

    Xeon2901: Hey just change the "configurable" to match your store ID and run the script. A few people using the PHPCLI version have made copies of the script and run it as a CRON job ($STOREID = '1';)

    Another method is to look at the first CLI version I wrote which uses ARGS from the command line.

    Glad to see its a real message; I saw the username in my email and expected to be sold viagra ;o)

  117. Hi,

    I have these three files used:

    cache.php

    cache2.php

    cache_vycisti.php

    These files are working properly for a Store View 1.

    Now I would like to delete from the Store View 2 cache.

    Which file should I change?

    How do I find the line about Store ID query?

    Thank you very much !

  118. Thank you so much for your script it has saved our lives with our websites all now have layered nav running 1.3.2.4 on our VSP 1and1 server, we still have no price attribute in the layered nav which is throwing up an error 500 when we attempt to turn on filterable with results, any ideas welcomed…

    Once again cheers very much Chris

  119. Magento 1.4 is still non-functional in three indexing areas (site with 30,000 products across about 20 categories):

    Index product and categories url rewrites

    Reorganize EAV product structure to flat structure

    ebuild Catalog product fulltext search index

    All of the above freeze in 'Processing' mode – presumably a server timeout (semi-dedicated at Gigapros who allow up to 10% CPU usage.

    Interestingly essentially the same site but with 60,000 products works fine on Magento 1.3 using Badger, but at a different shared host…

  120. Its an old saying but 'mileage may vary' seems to fit quite well here; badgers been tested against a product database of 32,000 products with over 100 attributes. Surprisingly without dying.

    Magento 1.4.1 must be tested on a hefty system, if you look at the Enterprise setup white-paper the MySQL set-up is probably the biggest set-up requirement with huge memory requirements advised to the potential users.

    If you trace what Magento actually does during indexing, it creates a large transaction in PHP by hammering the database then pushes this huge transaction (per product) to the database. Its an immense query and although they've optimized it a bit, its less than forgiving.

    Magento 1.1.4 handles post-indexed sites (I index on my over-specced workstation before pushing the database) better than its predecessor but changes to the code mean 1.1.4 fudges badger and I've not really had many requests to drag it on; which is either down to people no updating, or people not needing it, hard to say which.

    I've got a half eaten article about setup requirements for magento sat in my drafts-bin, with any luck I'll shed some light on the nicest looking hungry hungry hippo of an ecommerce platform available for free :o)

  121. Hello,

    I'm using a Magento 1.3.2.4 on a eCommerce site and I'm having problems about displaying the layered navigation.

    I'm using a attribute called "Object type" set as dropdown.

    That attribute is set as "Use In layered navigation" = filtered (I've tried either with "with results" and "no results")

    The attribute is in a attribute set.

    The elements of the subcategories that I want to show the layered navigation are all of the same attribute set.

    The elements of that subcategories have the attribute "Object type" always set (it's not left blank).

    On categories not set as anchor it shows the navigation menu displaying the sub-categories.

    On categories set as anchor it displays nothing.

    I've tried this script solutions but it doesn't solve my problem.

    Is it possible to make any echo of variables somewhere (or looking in the database) to find where's the problem?

  122. Regarding my previous message.

    The site I'm working on, is using a template that seems the default one but I've checked that has some hardcoded pieaces of code. One of them had comment out the code that echo the layered navigation search… so that was the problem ;)

Comments are closed.