Using IndexDen as a backend for SphinxSearch

R_J

Recently I stumbled upon IndexDen which is a commercial sphinx hoster and has a free plan for 15k documents. My first idea was to tweak your plugin in order to use IndexDen, so that smaller boards being on hosted services with no possibility to set a sphinx server of their own, could benefit from your work, too.

Looking at your plugin has impressed me: your work is far more complex than I expected! (I haven't used it by now)

So before I even started to dive into your code, I thought I'll simply ask you, @mcu_hq: do you think tweaking your plugin to use something like IndexDen as the search backend would be just a tweaking or would it be a major rewrite of what you've done?

hgtonight

This is a pretty cool service!

As far I can tell from the documentation FAQ, this service isn't well suited to a forum and wouldn't really overlap with the SphinxSearch plugin.

That said, I am pretty new to Sphinx and I hope I am wrong.

R_J

I haven't really looked at Sphinx Search by now and so I really know nothing about it (accept for its purpose). But that snippets from the example page made me hope its API may be similar to how you interact with Sphinx which would make it easy to "simply" change the backend of the SphinxSearch plugin:

$index->add_document($doc_id, array('text'=>$text,
    'title'=>$title,
    'author'=>$author));
...

$res = $index->search($query." author:$author");

mcu_hq

@R_J said:
So before I even started to dive into your code, I thought I'll simply ask you, mcu_hq: do you think tweaking your plugin to use something like IndexDen as the search backend would be just a tweaking or would it be a major rewrite of what you've done?

I looked at it for a few minutes and realized that IndexDen doesn't use a config file like sphinx does, but instead relies on your script to use batch indexing with an array. This is the part that worries me with a large set of documents since the script may timeout and such. It also seems tedious.

So to get the current plugin to work would require an overhaul of the config file. The SQL quireies would have to be run inside of the PHP file and then sent to IndexGen to be processed. After that, each of the widgets would need to change their sphinx API call slightly.

The biggest issue though is that IndexDen does not seem to support everything that a local installation of sphinx can do. For instance, wildcards, stopwords, distributed indexes, highlighting words, etc. If you look at the difference between IndexDen documnation and sphinx's documentation, you will see that IndexDen leaves out 95% +of the stuff mentioned in sphinx. I'm not saying that IndexDen is not a good alternative, but rather it is better suited for a "limited" sphinx experience, but I think that could be good enough for most people.

Would you mind first getting your plugin working with IndexDen? They have a free service with a limited amount of indexed docs that you can use.Then we can see how difficult it would be to port.

R_J

Thanks for your feedback!

My plugin? lol! I've started a search plugin several month ago, made it to create a search index for all discussions and comments and I think that the indexing is quite okay for what you can expect from a simple mysql solution but when it was time to implement some user interface, I dropped it. I suck at that. But I've put it on an ever growing list of plugins that I'd like to finish some day...

hgtonight

@R_J said:
...put it on an ever growing list of plugins that I'd like to finish some day...

You have truly become a developer. XD

R_J

So true! When I started here I was always eager to hear plugin ideas and now I would love some newbie would ask for some inspiration...