Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck

Solution 3 – Running the Layered Navigation Cache; PHP-CLI & Browser Version

Magento 1.4 will probably solve this issue for many people, so it may be worth giving it a squirt )

Fuck up central, exporting the script caused some of the comments to drop to multiple lines. If you had problems re-download please.
Browser edition now updated to actually delete files on completion and in clicking complete deleting the files and stopping the refreshes.
———————————-
Ok so I buggered up the first release (hence the post dissapearing), Basically I’d “counted” the products rather than grabbing the last ProductID as used by the Singleton method. So I’ve resolved that (thanks Mark H for pointing it out, I must of been half asleep). Processing through none existant records has no detrimental effect (it checks and jumps them) so I’ll skip on the suggestion of running through the array, thanks.

Well here goes, part three… Deep Breath.

First some specs:

On my test server (restrictions as on shared hosting in CHROOT);
100 records took this long

Previous 100 records in

#Record 101#CURRENT MEMRY USAGE = 152,411,480;
real    5m37.504s
user    1m19.869s
sys     0m3.362s

New 100 records in

#Record 101#CURRENT MEMRY USAGE = 40,411,480;
real    2m30.101s
user    0m34.841s
sys     0m0.907s

New Features

*NEW* PHP-CLI Edition
* Auto Product Count
* Auto On Completion Destroy Temp Files
* Various Optimizations

*NEW* PHP Web Browser Edition
* Three files;
** Wrapper (which you run from your browser)
** Two IFRAMES: The main functional script and a counter to show its not fallen asleep
** Auto Updating output; set off and sleep mode p

Now for the breakdown (script not mental)

Magento Layered Cache Fix PHP-CLI Edition

#!/usr/bin/php -q
<?php
//#######################################
//# Magento Layered Nav - Badger || Duck
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# MLBv3.1.1
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
try{
	ob_implicit_flush(true); //Saves having to flush manually
	if(!ini_get('safe_mode')){set_time_limit(0);}/*Set time limit to unlimited, though we shouldnt need to*/
	ignore_user_abort(); //Lets the script run to completion even if you kill the browser
 
	require_once("app/Mage.php");
	Mage::app('admin'); //as per magento handling
	Mage::setIsDeveloperMode(true); //apmh
	umask(0);
 
	/*{ CONFIGURABLES }*/
	$HM = 800; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
 
	/* Automatically Count Products */
		$total = NULL;
		if($HOWTORUN == 1)
		{
			/*## CLI Version */
			$prodCount = "cm_pd";
			if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
			else
			{
					$fh = fopen($prodCount, 'w') or die("can't open file");
 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
 
					$total = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
 
					fwrite($fh, $total);
					fclose($fh);
			}
			/*## End CLI Version*/
		}
		else
		{
			/*## Browser Version */
			if((!isset($_SESSION['pcount']))||($_SESSION['pcount']==NULL)){ 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$endValue[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
					$_SESSION['pcount'] = $endValue;
				$total = $_SESSION['pcount'];
			}
			else{ $total = $_SESSION['pcount']; }
			/*## End Browser Version */
		}
		if($total == NULL){ throw new Exception("Total Wasn't Captured"); }
	/* End Count Products */
 
	echo("n$totaln");
 
	$countFile = "cm_cnt";
	$count = null;
	$store = Mage::app()->getStore($STOREID);
 
	//Create a count file if we dont have one
	if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	else
	{
			$fh = fopen($countFile, 'w') or die("can't open file");
			$count = 0;
			if((int)$count == 0) $count = 1;
			fwrite($fh, $count);
			fclose($fh);
	}
 
	//If we havent started counting yet clear cache.
 
	if($count < 2)
	{
			Mage::app()->cleanCache();
			//Clear the database flag from any previous "normal" indexes that failed
			$flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
			if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) $flag->delete();
	}
 
	$max = ($count+$HM);
	$sx = Mage::app()->getStore('1');
 
	ob_start();
	echo "start loop (max = $max) (count = $count)";
 
	for($i = $count; $i <= $max; $i++)
	{
	ob_end_flush();
	ob_start();
				if($i == $total)
				{
						echo "<br><h1>Completed</h1><br>";
						@unlink($prodCount);
						@unlink($countFile);
					exit(1);
				}
 
 
			Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $sx);
 
			echo("n #Record $i#CURRENT MEMRY USAGE = ".memory_get_usage().";n"); //UPDATE SCREEN
 
			//Write where were up to
			$fh = fopen($countFile, 'w') or die("can't open file");
			fwrite($fh, $i);
			fclose($fh);
	ob_flush();
	flush();
	}
 
}catch(Exception $e){ echo"<br><p style=red>ERROR: $e</p><br>"; exit(1);}
#EOF
?>

Mostly though you’ll need to change this bit to run it in PHP-CLI over SSH…

/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/

How many records to process at once: I comfortably managed 500 at a time without it breaking a sweat
StoreID: I’ll let you guess, but for most of us mortals 1 is the default.
HowToRun: This is extendable so basically the guts is there to allow you to use Sessions (in the browser version) IO readings slightly faster so in the end it won over sessions

Thats pretty much it, aha, yep thats all, stop asking.

Simply run it from shell (or a script) as follows

time php -q -f cm_LayeredCache_badger.php

Time will output the time it took to run the script; -f is telling php to load the file; -q is quiet mode, but thats on by default.

Magento Layered Cache Fix Browser Edition

badger-cm_LayeredCache.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
try{
	ob_implicit_flush(true); //Saves having to flush manually
	if(!ini_get('safe_mode')){set_time_limit(0);}/*Set time limit to unlimited, though we shouldnt need to*/
	ignore_user_abort(); //Lets the script run to completion even if you kill the browser
 
	require_once("app/Mage.php");
	Mage::app('admin'); //as per magento handling
	Mage::setIsDeveloperMode(true); //apmh
	umask(0);
 
	/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/
 
	/* Automatically Count Products */
		$total = NULL;
		if($HOWTORUN == 1)
		{
			/*## CLI Version */
			$prodCount = "cm_pd";
			if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
			else
			{
					$fh = fopen($prodCount, 'w') or die("can't open file");
 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$total = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
 
					fwrite($fh, $total);
					fclose($fh);
			}
			/*## End CLI Version*/
		}
		else
		{
			/*## Browser Version */
			if((!isset($_SESSION['pcount']))||($_SESSION['pcount']==NULL)){ 
					$tmpIDs = Mage::getModel('catalog/product')->getCollection()->getAllIds();
					$_SESSION['pcount'] = $tmpIDs[sizeof($tmpIDs)-1]; //-1 Accounts for 0 in array.
				$total = $_SESSION['pcount'];
			}
			else{ $total = $_SESSION['pcount']; }
			/*## End Browser Version */
		}
		if($total == NULL){ throw new Exception("Total Wasn't Captured"); }
	/* End Count Products */
 
	echo("n$totaln");
 
	$countFile = "cm_cnt";
	$count = null;
	$store = Mage::app()->getStore($STOREID);
 
	//Create a count file if we dont have one
	if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	else
	{
			$fh = fopen($countFile, 'w') or die("can't open file");
			$count = 0;
			if((int)$count == 0) $count = 1;
			fwrite($fh, $count);
			fclose($fh);
	}
 
	//If we havent started counting yet clear cache.
 
	if($count < 2)
	{
			Mage::app()->cleanCache();
			//Clear the database flag from any previous "normal" indexes that failed
			$flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
			if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) $flag->delete();
	}
 
	$max = ($count+$HM);
	$sx = Mage::app()->getStore('1');
 
	ob_start();
	echo "start loop (max = $max) (count = $count)";
 
	for($i = $count; $i <= $max; $i++)
	{
	ob_end_flush();
	ob_start();
				if($i == $total)
				{
					echo "<br><h1>Completed</h1><br>";
					exit(1);
				}
			Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $sx);
 
			//Write where were up to
			$fh = fopen($countFile, 'w') or die("can't open file");
			fwrite($fh, $i);
			fclose($fh);
	ob_flush();
	flush();
	}
 
}catch(Exception $e){ echo"<br><p style=red>ERROR: $e</p><br>"; exit(1);}
#EOF
?>

badger-count.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck - Counter
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
?>
<html>
<head>
<?php if(!isset($_GET["done"])){ ?>
<meta http-equiv="refresh" content="5" />
<?php } ?>
</head>
<body>
<?php
$prodCount = "cm_pd";
$countFile = "cm_cnt";
 
 
if(isset($_GET["done"])){ 
	unlink($prodCount); 
	unlink($countFile);
	echo "Files Deleted";
	exit(1);
}
 
if((file_exists($prodCount))&&(filesize($prodCount)>1)){
							$fh = fopen($prodCount, 'r');
							$total = fread($fh,filesize($prodCount)) + 1;
							fclose($fh);
			}
if((file_exists($countFile))&&(filesize($countFile)>1)){
					$fh = fopen($countFile, 'r');
					$count = fread($fh,filesize($countFile)) + 1;
					fclose($fh);
					if((int)$count == 0) $count = 1;
	}
	if(isset($total)){
		$left = ($total - $count);
		echo "<b>[Count: $count | Total: $total][ $left to go]</b>";
				if($left == 0)
				{
					echo "<br /><a href='badger-count.php?done=1' target='_parent'>Click Here To Complete and Delete Count files</a>";
					exit(1);
				}
	}
?>
</body>
</html>

badger-wrapper.php

<?php
//#######################################
//# Magento Layered Nav - Badger || Duck - WRAPPER
//#######################################
//# @author: Chris McKee [chrismckee.co.uk]
//# @licence: GNU, basically do what you like but
//#           leave my name somewhere
//# @licenceurl: http://www.gnu.org/copyleft/gpl.html
//#######################################
?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-gb">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Magento Layered Cache</title>
<meta http-equiv="robots" content="noindex,noarchive" />
<style type="text/css">
<!--
@charset "utf-8";
/************************************************************
 * Core: Magento
 * @Target: Layered Nav refresh
 * @author: Chris.McKee
 * @date: 23-07-2009
************************************************************/
 
html{ font-size:16px;color:#000; background:#F9F9F9; } /* Set BASE Font Size */
 
/* Reset Browser Differences */
*,body,div,dl,dt,dd,ul,ol,li,h1,h2,h3,h4,h5,h6,pre,form,p,blockquote,th,td{margin:0;padding:0px;}
table{border-collapse:collapse;border-spacing:0}
address,caption,cite,code,dfn,em,th,var{font-style:normal;font-weight:400}
ol,ul,li{list-style:none}
caption,th{text-align:left;}
h1,h2,h3,h4,h5,h6{font-size:100%;font-weight:400}
q:before,q:after{content:''}
fieldset,img,abbr,acronym{border:none;font-variant:normal;}
sup{vertical-align:text-top;}
sub{vertical-align:text-bottom;}
input,textarea,select{font-family:inherit;font-size:inherit;font-weight:inherit;}
input,textarea,select{*font-size:100%;}
legend{color:#000;}
html, body { height:100%;	width:100%;	}
/*EORESET*/
/* Now thats over we can begin */
body {font:0.8em "Trebuchet MS", Helvetica, Verdana, Arial, FreeSans, sans-serif;}
h1{
	font-size:1.6em;
	position:absolute;
	left:4px;
	display:block;
	top: 8px;
}	
#refresh{
  position:absolute; 
	left:-1000px;
	height:0px;
	width:0px;
}
#counter{ position:relative; margin: 25% auto 0px; border:0px; outline:0px;}
#counter * {font-size:5em; text-align:center; }
small a{ position:absolute; bottom:8px; right:8px; display:block; text-decoration:none; color:#666; }
 
-->
</style>
<script type="text/javascript">
//<![CDATA[
//40 Seconds (no point in changin this)
var intervalID;
function startREFRESH(){  intervalID = setInterval(reloadREFRESH, 40000); }
function reloadREFRESH(){ refresh.location.reload(1); }
function stopREFRESH(){ clearInterval(intervalID);}
//]]>
</script>
<link rel="Shortcut Icon" type="image/x-icon" href="http://chrismckee.co.uk/favicon.ico" />
</head>
 
<body onload="startREFRESH();">
<h1>Layered Navigation Refresh From Browser</h1>
<iframe src="badger-count.php" id="counter" name="counter">You At Least Need a Browser that Supports IFRAMES</iframe>
<iframe src="badger-cm_LayeredCache.php" id="refresh" name="refresh" >You At Least Need a Browser that Supports IFRAMES</iframe>
<small><a href="http://chrismckee.co.uk/?s=layered+navigation">ChrisMckee Designs</a></small>
</body>
</html>

/*{ CONFIGURABLES }*/
	$HM = 100; /* How many records to process at once*/
	$STOREID = '1'; /* Theres many ways to skin a monkey but this will help reduce processing by reducing requests.*/
	$HOWTORUN = 1; /*1 = PHP CLI (aka SSH Mode), 2 = Browser (aka if your using the wrapper)*/

How many records to process at once: Dont Bother changing this
StoreID: I’ll let you guess, but for most of us mortals 1 is the default.
HOWTORUN: As with the CLI version you can ignore this

To run simply upload to the root of the magento site (as with all the other instructions) and open the badger-wrapper in your browser )

Let me know how you get on and and issues

Previous Versions

DOWNLOADS (updated 20090728)

20090728: Line: if($count == 0) changed to if($count < 2) otherwise condition never met; meaning if your indexes were running (on the cache page your "KILL" option was still there) the script would run, but MAGE would not store your indexed results.
PHPCLI Layered Cache – vbadger – Zipped PHP File
Browser Layered Cache – vbadger – v1.3.4

TROUBLESHOOTING BANG

Many people on shared hosting experience the following error

Failed loading /usr/local/Zend/lib/ZendExtensionManager_TS.so: /usr/local/Zend/lib/ZendExtensionManager_TS.so: cannot open shared object file: No such file or directory

In this case it’s apparent that your CHRooted/shared environment isn’t set to allow direct access to Zend as required by Magento; in this case its often a good idea to contact your Web Host to see if this is something they can resolve for you.

To show its not this script that causes the issue try the following standard import of Mage…

<?php
//Create a file called testmage.php and add the following code and run. php -qf testmage.php via SSH
//Errors don't always appear in shared environments when using a browser.
require_once("app/Mage.php");
?>

The only “solution” is to run the browser version which acts more like Magento does, though with this mileage may vary

Thanks to Oleksiy for letting me into his server to diagnose this issue.

#2 The indexing finishes but my layered navigation doesn’t appear: I’ve heard this one 5 times now and happily its always the same problem… It’s not appeared because you’ve not turned it on.

Required attribute conditions to be shown on layered navigation filters:

Attribute properties / Catalog Input Type for Store Owner = Dropdown or Multiple Select (or the System Attribute, Price)
Frontend properties / Use In Layered Navigation = Filtered *
System properties / Data Type for Saving in Database = Integer
System properties / Globally Editable = Yes
Please note that a category should have the ‘Is Anchor’ property set to Yes in order for the layered navigation block to show.

Source: http://www.magentocommerce.com/wiki/general/layered-navigation

166 thoughts on “Magento Commerce – Layered Navigation Cache Error – Part Badger Part Duck”

Riccardo May 30, 2010 at 10:29

Regarding my previous message.

The site I'm working on, is using a template that seems the default one but I've checked that has some hardcoded pieaces of code. One of them had comment out the code that echo the layered navigation search... so that was the problem
1. Chris McKee Post authorMay 30, 2010 at 12:43
  
  Good stuff, ghosts in the machine tend to be hard for me to guess p
Riccardo May 26, 2010 at 10:33

Hello,

I'm using a Magento 1.3.2.4 on a eCommerce site and I'm having problems about displaying the layered navigation.

I'm using a attribute called "Object type" set as dropdown.

That attribute is set as "Use In layered navigation" = filtered (I've tried either with "with results" and "no results")

The attribute is in a attribute set.

The elements of the subcategories that I want to show the layered navigation are all of the same attribute set.

The elements of that subcategories have the attribute "Object type" always set (it's not left blank).

On categories not set as anchor it shows the navigation menu displaying the sub-categories.

On categories set as anchor it displays nothing.

I've tried this script solutions but it doesn't solve my problem.

Is it possible to make any echo of variables somewhere (or looking in the database) to find where's the problem?
Chris McKee Post authorMay 10, 2010 at 17:31

Its an old saying but 'mileage may vary' seems to fit quite well here; badgers been tested against a product database of 32,000 products with over 100 attributes. Surprisingly without dying.

Magento 1.4.1 must be tested on a hefty system, if you look at the Enterprise setup white-paper the MySQL set-up is probably the biggest set-up requirement with huge memory requirements advised to the potential users.

If you trace what Magento actually does during indexing, it creates a large transaction in PHP by hammering the database then pushes this huge transaction (per product) to the database. Its an immense query and although they've optimized it a bit, its less than forgiving.

Magento 1.1.4 handles post-indexed sites (I index on my over-specced workstation before pushing the database) better than its predecessor but changes to the code mean 1.1.4 fudges badger and I've not really had many requests to drag it on; which is either down to people no updating, or people not needing it, hard to say which.

I've got a half eaten article about setup requirements for magento sat in my drafts-bin, with any luck I'll shed some light on the nicest looking hungry hungry hippo of an ecommerce platform available for free )
Simon Verdon May 10, 2010 at 17:09

Magento 1.4 is still non-functional in three indexing areas (site with 30,000 products across about 20 categories):

Index product and categories url rewrites

Reorganize EAV product structure to flat structure

ebuild Catalog product fulltext search index

All of the above freeze in 'Processing' mode - presumably a server timeout (semi-dedicated at Gigapros who allow up to 10% CPU usage.

Interestingly essentially the same site but with 60,000 products works fine on Magento 1.3 using Badger, but at a different shared host...
Tim Clark April 28, 2010 at 23:00

Thank you so much for your script it has saved our lives with our websites all now have layered nav running 1.3.2.4 on our VSP 1and1 server, we still have no price attribute in the layered nav which is throwing up an error 500 when we attempt to turn on filterable with results, any ideas welcomed...

Once again cheers very much Chris
Xeon2901 April 26, 2010 at 16:40

Hi,

I have these three files used:

cache.php

cache2.php

cache_vycisti.php

These files are working properly for a Store View 1.

Now I would like to delete from the Store View 2 cache.

Which file should I change?

How do I find the line about Store ID query?

Thank you very much !
Chris McKee Post authorApril 25, 2010 at 17:11

MagicCardz, James: This script system is written purely for Versions below 1.4; the changes they made to the system (they pretty much rewrote the indexing), breaks this script like a twig. I'll inevitably end up rebuilding it as some people still have issues on 1.4.1 with various bits of indexing. The bad news is magento couldnt give a rats arse about people hosting on servers with less than 256 mb of memory available for PHP, which is plain fucking stupid.

Xeon2901: Hey just change the "configurable" to match your store ID and run the script. A few people using the PHPCLI version have made copies of the script and run it as a CRON job ($STOREID = '1';)

Another method is to look at the first CLI version I wrote which uses ARGS from the command line.

Glad to see its a real message; I saw the username in my email and expected to be sold viagra ;o)