Magento Commerce – Layered Navigation Cache Error

Part Deux Solves many of the issues found with this script

Updated 19-03-09

Correct as of 11 March 2009 Hopefully in the near future Varien (behind the Magento) will pull the proverbial finger out and fix this most irritating of bugs.
Bug Traq: http://www.magentocommerce.com/bug-tracking/issue?issue=5424

Problem

When running a cache refresh on Layered Navigation the script times out producing either an Error 500 or server based error. The problem is then made worse by magento’s failure to recognise that the process didn’t complete.
Magento fails to update the running flag on the core-flag index and as such informs the user that the script is still running, and at no point will allow the script to be killed via the admin, as the process has already been killed by the server.

To be fair we must add a few prerequisites that add to this problem…

Problem is caused by the complexity of the query being run
Problem is more apparent on Shared hosting then on dedicated (as you can set a script execution limit)
To solve the problem the developer needs to check (using phpinfo() ) that the server is not running in safe-mode (this is for fix 1)
If the server is running in safe mode the execution time of the script is FIXED and therefore the developer will need to run the script in sets based on the number of products to be indexed.

Solution 1 – Running the Layered Navigation Cache from PHP-CLI

Disable the indexer before we do anything else

The indexer is a pain in the arse, that with a bit of thinking would of worked fine, shame really. Anyhoo, heres how we break it so it stops breaking our sites.

~~Go to /app/code/core/Mage/CatalogIndex/Model/ and open Indexer.php for editing in your favourite app (I like netbeans)~~
~~Done, good, now slap that up to your server. When the "Layered Navigation – Cache Control (beta)" tries to refresh it will just return that its done and leave all of our hard work alone. YEY~~
~~Rest your fingers before the next bit, typing that ridiculously long folder url must of been tiring~~
Sod all that editing, simply turn it off in the admin, and leave indexer.php alone.

Around line 282 (function plainReIndexer) I removed the following

        if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) {
            return $this;
        } else {
            $flag->setState(Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING)->save();
        }

This stops the swine creating the flag to say that the process is running and ruining our fun. Make a note of this, or if your using SVN, happy dont give a crap as yours is logged for you.

Create the PHP File via SSH

First of all go to the root of your magento site, this needs to be the root as the file paths used here are based on the scripts ability to see the app/ folder.

Create a file, call it what you want, but for the sake of this example it will be cm_cache.php
To do this you can use vi (standard on linux/unix servers); so type
vi cm_cache.php
vi will open in a new document, for now press escape then type
:wq
vi will save and exit
Type
chmod 755 cm_cache.php
and hit enter, this sets cm_cache to executable
now lets ram some content into cm_cache.php (skip this if you’ve downloaded/edited mine from below)
copy the following code and vi cm_cache.php again and this time on the VI edit screen press a – the bottom of the screen (left) should say INSERT, which unsurprisingly means your in insert mode, press SHIFT INSERT to paste.

Code:cm_cache.php

    #!/usr/bin/php -q
    //################################
    //# Magento Layered Nav QuickFix
    //################################
    //# @author: Chris McKee
    //# @licence: GNU, basically do what you like but
    //#               leave my name in.
    //################################
 
    set_time_limit(0);
    ignore_user_abort();
 
    $mageFilename = 'app/Mage.php';
    require_once $mageFilename;
    umask(0);
 
    // clean overall cache
    Mage::app()->cleanCache();
    echo "Cache Cleaned!n";
    flush();
 
    $opt;
    $count = 0;
    foreach ($_SERVER['argv'] as $arg)
    {
      $opt[$count++] = $arg;
    }
 
    #1 start #2 end;

    $start = $opt[1];
    $end = $opt[2];
    //The Damn Loop
    $c = 0;
    // Only process if a storeid is passed
        $store = Mage::app()->getStore('1');
        for($i = $start; $i < $end; $i++) //for i=start-record; i < last-record-id; ...
        {
            Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $store);
            echo " . ";
    // PRINT CURRENT MEMRY USAGE:
    print "CURRENT MEMRY USAGE = ".memory_get_usage()."-";
 
    // PRINT CURRENT MEMRY LIMIT:
    print "/MEMRY LIMIT = ".ini_get('memory_limit')."n";
 
      $c++;
            @ob_flush(); flush();
            sleep(1);
        }
    echo "count: $c";
    flush();
    ?>

Now exit by pressing escape, then typing :wq (: = menu, w = write, q = quit)
Lastly back in your ssh shell type
php -q -f cm_cache.php 2 750
Replace the numbers startindex endindex, I just used the max id from the products index in the admin and the lowest as the start index. Hit enter and depending on how many records you have, this could take a few minutes to an hour.

Solution 2: When you’ve already set it going or when the shit hits the fan

Firstly lets clear the crap indexes out of our database, by pushing a bit of SQL through MySQL…

TRUNCATE `catalogindex_eav` ; #Cleans-3-indexes
TRUNCATE `catalogindex_minimal_price` ;
TRUNCATE `catalogindex_price` ;
TRUNCATE `core_flag` ;#Then-Cleans-The-Currently-Caching-Flag

That should get rid of the process from the database, and put you back where you started.

*** Note *** This isn’t complete, I will add a database check to this to do solution 2 before the first part; create a wrapper script to calculate the max/min values and provide a non ssh root to do this. In fact I’ve already written it all I need to upload it all and run xdebug for a while. )

16 thoughts on “Magento Commerce – Layered Navigation Cache Error”

Chris McKee Post authorJuly 6, 2009 at 21:43

@jk Good idea; I was planning on having an updated version out a few weeks ago, I'm a bit stuck for time due to other things in the pipeline.

Try and remember when modifying this file that initially it was written to be run from SSH to minimise the risk of cancellation and to reduce the memory consumption. The more text you write to the screen via echo, regardless of how nice verbose is, will increase the buffer and reduce the number you can process in each run.

@spatter: thanks more and more to come. Unless I get hit by a bus that is.

Ta

Chris
JK July 5, 2009 at 20:34

Hi, my friend ask my to repair his shop because he got problems with his layered navigation in Magento store.

I was searching over the internet and I found one FIX created by Chris McKee. THX Chris

His script was perfect but some automatic functions were missing, therefore I’ve created my own script.

Check it here: <a href="http://jksite.com/blog/2009/07/magento-layered-navigation-cache-error" target="_blank">http://jksite.com/blog/2009/07/magento-layered-na...
Many THX Chris!
spatter July 5, 2009 at 17:38

I've had your blog in my bookmarks for a couple of days now and just wanted to say that I really love your articles! ^^
JK June 25, 2009 at 13:24

Hello Chris, Perfect !! It is working for my shop (id 3) where I have 117 products, but if I do it for a shop with 1077 products (id 5) it is not working everytime some error,I was thinking that this file should be able to make it one by one... without execution time is exceeded, if I select ony 1 as $hm, not working, max execution time 150 s exceeded...

Could you make any chnages to the script to make it working ?

THANKS
Chris McKee Post authorMay 25, 2009 at 23:28

@Scott: Generally, yes you should use local folder with a copy of the structure + the file you wish to override to overload that chunk of code; In cases such as this where I'm basically commenting something out to fix an issue I prefer to leave the files as is and change the core.

As I use SVN + Patch I can see when alterations have been made to this file and if I see that its beneficial to the file I can accept their changes over mine and "in pigs can fly" world stop using my hack to handle the layered nav. If I was using the Local file I would have to make sure I was copying my files across every time there was another change to the file.

The Inchoo article makes sense for new users but if your comfortable enough to manage your work in SVN, using the downloader so you can upgrade automatically is a lot like giving away any control you have.
Scott May 24, 2009 at 19:50

One other note worthy mention is never comment out core code. Put an exact copy of that file into the same folder structure under local then when you upgrade it wont overwrite your hard work. Read this post if you dont believe me.
http://preview.tinyurl.com/pfl8el
Scott May 24, 2009 at 19:47

@Dennis, also look into servint, they have been our host through our magento install and have happily made any changes and quickly that we needed for magento. IE: Innodb tables, php soap etc.
Dennis Crawford May 15, 2009 at 17:26

Chris,

Thanks for the help you have given me, it appears I have bigger issues than worrying about this right now. Obviously GoDaddy is not the correct host to use for Magento. I am looking into a couple of other providers with SimpleHelix being the forerunner.

Dennis
1. Chris McKee Post authorMay 15, 2009 at 20:06
  
  @Dennis:
  Ideally you need a decent host (preferably one that actually says its a magento commerce host; as it means they've tested it) and in an ideal world MOD-PHP (Mod-apache-php) rather then PHP-CGI (FASTCGI) Which is usually the choice on shared hosts.
  
  So I'd say either united hosting who are the only big shared hosting providers that have proven to be good. Or a dedicated server with someone like Aplus.net or Tagadab; in which case you'll have to look at getting your server scanned for PCI Compliance (UK) or the US Equivalent.
Dennis Crawford May 11, 2009 at 10:34

Chris,

I have followed your directions to a "T". File permissions, downloading, creating an empty CNT file. If I try to run this from putty using php -f cm_cachephp.php I still get the same error regarding the store ID.

I have checked in ADMIN and it is in fact 1. When I run this inside Firefox this is what I get; this output is generally the same but will vary depending on how many products I set to process. The page stops there and a refresh will result in the same message as below but continue with the left number sequence. I have set the products to 265 which is what we currently have. If I continue refreshing the page it will keep going and going way past my current product count.

#1#CURRENT MEMRY USAGE = 4951504;

#2#CURRENT MEMRY USAGE = 4951704;

#3#CURRENT MEMRY USAGE = 4951720;

#4#CURRENT MEMRY USAGE = 4951720;

#5#CURRENT MEMRY USAGE = 4951720;

#6#CURRENT MEMRY USAGE = 4951720;
Chris McKee Post authorMay 11, 2009 at 00:15

@Dennis Crawford Line 16 is simply setting a variable; did you download the file or copy-paste the file which would only make sense if the contents of the file wasn't right.

<pre lang="php">$store = Mage::app()->getStore('1');</pre>

Make sure that get store is getting the right store number; it should be 1 if you only have one store.

Make sure the script is set to CHMOD 755 and if your systems a bit wierd (as a few hosts have proven to be) create a file called CNT and set that to 777.
Dennis Crawford May 10, 2009 at 23:04

I did now I am getting the same error in different line;

<pre class="bash">

syntax error, unexpected T_OBJECT_OPERATOR in

/home/content/c/e/s/cesavings/html/store/cm_cachephp.php on line 16

</pre>

Thanks for the help

Dennis
Chris McKee Post authorMay 7, 2009 at 22:06

@Dennis Crawford use the updated script http://chrismckee.co.uk/magento-commerce-layered-...
Dennis Crawford May 7, 2009 at 13:57

I am getting Parse Error-Syntax Error, unexpected T_OBJECT_OPERATOR in /url/cm_cache.php line 19
Chris McKee Post authorMarch 19, 2009 at 04:16

Hi Thad,

1. The memory consumption rises like nothing else; this is partly because of the complexity of the query being run and partly because Magento is bloated. As you'll note I have added the post to the bug log.

2. Feel free; I'm delaying the script to provide a breather between the pile of processes Mage runs through zend to create the database entries. If you watch the database tables, catalogindex_eav _minimumprice and _price you'll see that it adds thousands of records.

3. Ignore that, The fix it provides is just circumstantial. Its better to just go into the admin and turn it off under layered navigation cache. I'll change the post to reflect the changes I've made.

4. It's easier to turn it off in the admin... Magento doesn't seem to check... granted working with magento codes a lot like finding a needle in a haystack.

>> http://chrismckee.co.uk/magento-commerce-layered-...

Thanks

Chris
Thad March 18, 2009 at 19:15

I have a few questions:

1. The memory consumption of the script is really high. I'm on less than 500 products and it's over 140MB.

2. Also I'm wondering why the sleep() function is there. Without it it doesn't seem to work and the memory consumption never changes. Just wondering. Could that be adjusted to half a second with usleep()?

3. And if you could explain how the indexer is disabled by what you did. That would help.

4. One more thing. You can use custom code to disable that indexer by rewriting it in local.xml and creating your own model. Although I can't seem to get this to work right now. I will tell you when I figure it out.

I hope this solution works for me, I've been working on this for a while now.

Comments are closed.