Bayesian classifier for TTRSS

[fox]

there's three classifier categories now so after updating to https://github.com/gothfox/Tiny-Tiny-RS ... 878231a17b you all will need to do the following:

Code: Select all

delete from  ttrss_plugin_af_sort_bayes_categories

and reload tt-rss

[nameless]

Code: Select all

E_USER_ERROR (256)   classes/db/pgsql.php:46   Query INSERT INTO ttrss_plugin_af_sort_bayes_categories (category, owner_uid) VALUES ('NEUTRAL', 1) failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist LINE 1: INSERT INTO ttrss_plugin_af_sort_bayes_categories (category,... ^   admin   18:00
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query INSERT INTO ttrss_plugin_af_sort_bayes_categories (category, owner_uid) VALUES ('GOOD', 1) failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist LINE 1: INSERT INTO ttrss_plugin_af_sort_bayes_categories (category,... ^   admin   18:00
E_WARNING (2)   classes/db/pgsql.php:57   pg_num_rows() expects parameter 1 to be resource, boolean given   admin   18:00
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query SELECT id FROM ttrss_plugin_af_sort_bayes_categories WHERE owner_uid = 1 LIMIT 1 failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist LINE 1: SELECT id FROM ttrss_plugin_af_sort_bayes_categories WHERE o... ^   admin   18:00
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query CREATE TABLE IF NOT EXISTS ttrss_plugin_af_sort_bayes_wordfreqs ( word varchar(100) NOT NULL DEFAULT '', category_id INTEGER NOT NULL REFERENCES ttrss_plugin_af_sort_bayes_categories(id) ON DELETE CASCADE, owner_uid INTEGER NOT NULL REFERENCES ttrss_users(id) ON DELETE CASCADE, count BIGINT NOT NULL DEFAULT '0') failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist   admin   18:00
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query CREATE TABLE IF NOT EXISTS ttrss_plugin_af_sort_bayes_references ( id SERIAL NOT NULL PRIMARY KEY, document_id VARCHAR(255) NOT NULL, category_id INTEGER NOT NULL REFERENCES ttrss_plugin_af_sort_bayes_categories(id) ON DELETE CASCADE, owner_uid INTEGER NOT NULL REFERENCES ttrss_users(id) ON DELETE CASCADE, content text NOT NULL) failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist   admin   18:00
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query CREATE TABLE IF NOT EXISTS ttrss_plugin_af_sort_bayes_categories ( id SERIAL NOT NULL PRIMARY KEY, category varchar(100) NOT NULL DEFAULT '', probability DOUBLE NOT NULL DEFAULT '0', owner_uid INTEGER NOT NULL REFERENCES ttrss_users(id) ON DELETE CASCADE, word_count BIGINT NOT NULL DEFAULT '0') failed: ERROR: type "double" does not exist LINE 4: probability DOUBLE NOT NULL DEFAULT '0', ^   admin   18:00
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query INSERT INTO ttrss_plugin_af_sort_bayes_categories (category, owner_uid) VALUES ('NEUTRAL', 1) failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist LINE 1: INSERT INTO ttrss_plugin_af_sort_bayes_categories (category,... ^   admin   17:59
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query INSERT INTO ttrss_plugin_af_sort_bayes_categories (category, owner_uid) VALUES ('GOOD', 1) failed: ERROR: relation "ttrss_plugin_af_sort_bayes_categories" does not exist LINE 1: INSERT INTO ttrss_plugin_af_sort_bayes_categories (category,... ^

so ttrss failed to create ttrss_plugin_af_sort_bayes_categories in the first place!?
i will take a closer look when i get off work.

[fox]

plugin should create the tables when it loads, strange.

[nameless]

it is.
can anybody confirm this as a bug or did i fuck up my postgres setup?

[fox]

try updating to latest trunk, it's very unlikely that something is wrong with your database

i mean you can always disable the plugin and it'll go back to normal

[nameless]

I just updated to latest to latest trunk things are still broken though.
Also disabling the plugin doesen't revert things back to normal, errors still occur.

[fox]

if you disable the plugin while updates are ongoing you are going to continue having errors until all ttrss processes terminate

i just checked on mysql too and table was created automatically btw

e: also your log doesn't show anything about plugin failing to create the table, only that relation doesn't exist, strange

[nameless]

so it looks as if i broke my database

[fox]

like I said it is very unlikely, i'd even say impossible

[nameless]

mhm.
is there anything else i can provide that could help debug this?

[fox]

oh wait there was a typo in the psql schema script, update and it should fix itself

e: there was another thing in there, gonna be fixed in a few minutes

not the smoothest rollout so far

e2: done

[nameless]

yep fixed.
i am left with this error message now though

Code: Select all

E_WARNING (2)   classes/db/pgsql.php:38   pg_query(): Query failed: ERROR: null value in column "content" violates not-null constraint DETAIL: Failing row contains (1, SHA1:43cdfc3ce412d0b85cc3e371a358936edf6cf145, 1, 1, null).   admin   18:46
E_USER_ERROR (256)   classes/db/pgsql.php:46   Query INSERT INTO ttrss_plugin_af_sort_bayes_references (document_id, category_id, owner_uid) VALUES ('SHA1:43cdfc3ce412d0b85cc3e371a358936edf6cf145', '1', 1) failed: ERROR: null value in column "content" violates not-null constraint DETAIL: Failing row contains (1, SHA1:43cdfc3ce412d0b85cc3e371a358936edf6cf145, 1, 1, null).   admin   18:46

[fox]

you can just run

Code: Select all

alter table ttrss_plugin_af_sort_bayes_references drop column content

it was that thing i just fixed

e3: just a footnote, plugin starts automatic classification when ugly aka neutral category reaches 10k words stored

[fox]

i only have a few articles in BAD category and i'm seeing it match way too much, maybe its more emphasized because there's not much data so every word counts more or whatever

maybe limiting to two categories would work better, idk

[himynameschris]

I would say that not having enough articles trained and then used to update the model is going to be the biggest reason for your results, you should try with 10 or 20 in each category.

I've also noticed that the plugin is only removing the words "'the', 'that', 'you', 'for', 'and'", this list of stopwords (words that don't carry enough meaning to be included in Bayesian classification) could be expanded to include other common words.

For example, I've noticed that some RSS feeds such as those from Ars Technica, Forbes and BBS will contain text such as "Read more" or "Continue reading", or even "this is an article about...", these alone could throw your results off. I think this would greatly improve any model based on RSS feeds.