I expected a toy project, but it is a usable library, which required a lot of work. Good job on delivering. A few comments:
After reading "composer.json", I thought that the tests used a custom framework. I'm glad the project does not suffer from NIH syndrome, but the dev dependency on PHPUnit should be declared.
There should a warning that it's only meant for some Western Latin languages. The normalization of the input is built on a character table for a handful of cases. That's not enough for some Latin languages, e.g. Turkish. And any input with Cyrillic, Arabic, CJK and so on, will be ignored.
There is no Unicode normalization or cleanup. Real-life input have many corner cases, e.g. diacritics next to the characters, or invisible characters inside a word to prevent hyphenation. Unless I'm mistaken, this engine would treat the NFD form "fête" as "fe te", instead of the expected "fete", which the NFKD form "fête" produces. I suggest using ext-intl for Unicode normalization, at least as an option.
Lastly, I can't think of a use case for this library. I've always had access to some external service (MySQL, Postgresql, Manticore Search, Solr, etc.) or to a PHP extension for a local Sqlite with FTS. Even for hobby projects, I haven't deployed to a shared hosting for more than two decades.
Loupe is a great project and more feature-rich than php-fts (stemming, geo, Damerau-Levenshtein typo tolerance). The dependency difference isn't just about Composer packages. Loupe requires pdo_sqlite and SQLite >= 3.35.0, which isn't guaranteed on shared hosting.
Great tool. Does it work with german umlaut (äöü)?
I find it very useful because shared hosting is still big for me. I use ultra cheap shared hosting for nearly everything. No Server maintainance and no funky serverless stuff
An alternative for PHP full text search is the PHP DuckDB client [1], and using DuckDB with the Full-Text Search extension [2]. It also stores everything on the local file system and can scale to millions of records.
Thanks for the suggestion. DuckDB is a great tool, but it requires installing a native extension, which is exactly the scenario php-fts is designed to avoid. The primary target is shared hosting (OVH, Infomaniak, o2switch…) where you have no control over installed extensions. If you can install DuckDB, you can probably also run Meilisearch or Typesense, which are purpose-built for search.
18 comments
[ 3.0 ms ] story [ 47.1 ms ] threadAfter reading "composer.json", I thought that the tests used a custom framework. I'm glad the project does not suffer from NIH syndrome, but the dev dependency on PHPUnit should be declared.
There should a warning that it's only meant for some Western Latin languages. The normalization of the input is built on a character table for a handful of cases. That's not enough for some Latin languages, e.g. Turkish. And any input with Cyrillic, Arabic, CJK and so on, will be ignored.
There is no Unicode normalization or cleanup. Real-life input have many corner cases, e.g. diacritics next to the characters, or invisible characters inside a word to prevent hyphenation. Unless I'm mistaken, this engine would treat the NFD form "fête" as "fe te", instead of the expected "fete", which the NFKD form "fête" produces. I suggest using ext-intl for Unicode normalization, at least as an option.
Lastly, I can't think of a use case for this library. I've always had access to some external service (MySQL, Postgresql, Manticore Search, Solr, etc.) or to a PHP extension for a local Sqlite with FTS. Even for hobby projects, I haven't deployed to a shared hosting for more than two decades.
I built a ChatGPT/claude history search tool and it works surprisingly well.
There are other tools also. Not to detract from this tool but just to inform people about alternatives.
[1] https://duckdb.org/docs/lts/clients/php [2] https://duckdb.org/docs/current/core_extensions/full_text_se...