Sphinx Plugin Cron Job Errors

meshugy

Hello,

I followed the instructions included with this plugin and setup three cron job (main, delta, and stats.) They are running several times a day, but when I look at the log it's showing these errors for every attempt:

WARNING: failed to open pid_file '/var/log/searchd.pid'.
WARNING: indices NOT rotated.

Any idea what I can do to avoid these errors? As it stands, they are preventing the indexer from running. I can get it to work manually, but I'd prefer to have the cron jobs automating this task.

The FAQ with the plugin mentions turning off searchd if these errors occur. I can do that when manually running the indecer, but how do you do that with a cron job?

Thanks!

Michael

mcu_hq

I suspect it may have to do with permissions. Since it works fine when you manually do it, can you verify that the cron job is running with root privleldges or at least the same user level as when you manually index? For sanity, also check that the cron file has the right locations of the pid file.

meshugy

Ok, I set the cron jobs to run as sudo and it seems to be working now. Here is what my log file looks like now. Does this look correct?

Only the Delta and Stats have been indexed so far...the Main won't be attempted till tonight.

Sphinx 2.1.4-release (rel21-r4421)
Copyright (c) 2001-2013, Andrew Aksyonoff
Copyright (c) 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'...
WARNING: key 'port' is deprecated in /usr/local/etc/sphinx.conf line 237; use 'listen' instead.
indexing index 'vss_delta'...
Sphinx 2.1.4-release (rel21-r4421)
Copyright (c) 2001-2013, Andrew Aksyonoff
Copyright (c) 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'...
WARNING: key 'port' is deprecated in /usr/local/etc/sphinx.conf line 237; use 'listen' instead.
indexing index 'vss_stats'...
collected 2567 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 2567 docs, 19288 bytes
total 0.046 sec, 412242 bytes/sec, 54864.49 docs/sec
total 3 reads, 0.000 sec, 20.1 kb/call avg, 0.0 msec/call avg
total 10 writes, 0.000 sec, 11.9 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=22482).
collected 10 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 10 docs, 4713 bytes
total 0.111 sec, 42144 bytes/sec, 89.42 docs/sec
total 5 reads, 0.000 sec, 1.0 kb/call avg, 0.0 msec/call avg
total 10 writes, 0.000 sec, 1.3 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=22482).

mcu_hq

Looks about right. Best way to tell is to create new content and then let cron run. Search for the new material after cron ran to see if it got indexed.

meshugy

@mcu_hq said:
Looks about right. Best way to tell is to create new content and then let cron run. Search for the new material after cron ran to see if it got indexed.

It seems to be running the cron files correctly, but when I search for new content it's not indexed. So somehow it's still not indexing correctly. If I manually use the command

sudo indexer --all --rotate

It indexes everything properly and everything is searchable right away. Any ideas what may be preventing the cron from working correctly?

Here is my latest log file:

Sphinx 2.1.4-release (rel21-r4421)
Copyright (c) 2001-2013, Andrew Aksyonoff
Copyright (c) 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'...
WARNING: key 'port' is deprecated in /usr/local/etc/sphinx.conf line 237; use 'listen' instead.
indexing index 'vss_main'...
collected 66465 docs, 33.9 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 6.2 Mhits, 100.0% done
total 66465 docs, 33899649 bytes
total 9.308 sec, 3641840 bytes/sec, 7140.33 docs/sec
total 8 reads, 0.022 sec, 2473.0 kb/call avg, 2.7 msec/call avg
total 52 writes, 0.101 sec, 877.2 kb/call avg, 1.9 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=22482).
Sphinx 2.1.4-release (rel21-r4421)
Copyright (c) 2001-2013, Andrew Aksyonoff
Copyright (c) 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'...
WARNING: key 'port' is deprecated in /usr/local/etc/sphinx.conf line 237; use 'listen' instead.
indexing index 'vss_stats'...
collected 3030 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 3030 docs, 23001 bytes
total 0.082 sec, 279029 bytes/sec, 36757.56 docs/sec
total 3 reads, 0.000 sec, 23.8 kb/call avg, 0.0 msec/call avg
total 10 writes, 0.000 sec, 14.1 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=22482).
Sphinx 2.1.4-release (rel21-r4421)
Copyright (c) 2001-2013, Andrew Aksyonoff
Copyright (c) 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/usr/local/etc/sphinx.conf'...
WARNING: key 'port' is deprecated in /usr/local/etc/sphinx.conf line 237; use 'listen' instead.
indexing index 'vss_delta'...
collected 2 docs, 0.0 MB
collected 0 attr values
sorted 0.0 Mvalues, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 2 docs, 503 bytes
total 0.261 sec, 1922 bytes/sec, 7.64 docs/sec
total 5 reads, 0.000 sec, 0.1 kb/call avg, 0.0 msec/call avg
total 10 writes, 0.000 sec, 0.2 kb/call avg, 0.0 msec/call avg
rotating indices: successfully sent SIGHUP to searchd (pid=22482).

mcu_hq

That Looks alright to me. If you compare the cron output to the manual method output, are there any differences?

meshugy

@mcu_hq there is no entry in the sphinx cron log when I just run the indexer manually.

meshugy

OK, i figured it out. The path to the vss files was not set correctly in the sphinx.conf file. For example, the path to the vss_main was set as

index vss_main
{
    source          = vss_main_comment
    source          = vss_main_discussion
    path            =  vss_main

For the cron file to work, it had to be set to:

index vss_main
{
    source          = vss_main_comment
    source          = vss_main_discussion
    path            = /home/myusername/vss_main

I set the vss_delta and vss_stats the same way and now everything is getting indexed via the cron file.

mcu_hq

Ahhhhh. Thanks for tracking it down. That should have been replaced by the installer. I'll upload a fix on github!

meshugy

@mcu_hq said:
Ahhhhh. Thanks for tracking it down. That should have been replaced by the installer. I'll upload a fix on github!

My pleasure, thanks for the awsome plugin!