text is `Live concert recorded at BBC-Radio 1\x02\x02 on November 21, 2002. Plus rare B-sides and acoustic tracks.\r`
2019-08-02 21404, 2019
alastairp
(contains \x02)
2019-08-02 21410, 2019
ruaok
!m alastairp
2019-08-02 21410, 2019
BrainzBot
You're doing good work, alastairp!
2019-08-02 21426, 2019
alastairp
phew
2019-08-02 21430, 2019
alastairp
wasn't where we thought it was
2019-08-02 21412, 2019
ruaok
was this data that was just added to the DB/index, alastairp ?
2019-08-02 21436, 2019
alastairp
id | created
2019-08-02 21436, 2019
alastairp
--------+-------------------------------
2019-08-02 21436, 2019
alastairp
249425 | 2007-12-14 18:00:59.638629+01
2019-08-02 21442, 2019
pristine__
okay.
2019-08-02 21451, 2019
pristine__
thanks :)
2019-08-02 21452, 2019
ruaok
pristine__: while in our case we're not dealing with floating point numbers, we're dealing with tricky corner cases.
2019-08-02 21425, 2019
ruaok
in this case the increment for month ( +=1 ) is easy, but a month has different days. that leads to nasty edge conditions.
2019-08-02 21455, 2019
ruaok
if instead you go to day 1 of the month and then add 32 days and go back to day 1, you are guaranteed to get the right next first day of the month.
2019-08-02 21409, 2019
ruaok
pristine__: does that makes sense?
2019-08-02 21416, 2019
alastairp
there's something funky going on with the multiprocessing queue, my feeling is that the exception causes a sentinel to be added to the queue which either causes the solr writer to stop processing items from the queue, or it crashes
2019-08-02 21436, 2019
alastairp
which results in it not adding anything to solr after it gets this exception
2019-08-02 21440, 2019
pristine__
what if I go to day 1 of the month and then add 7 ?
2019-08-02 21459, 2019
pristine__
I am still in the same month, no ?
2019-08-02 21430, 2019
pristine__
(lemme know when it gets annoying, I hope I am not asking stupid questions :( )
2019-08-02 21428, 2019
ruaok
correct.
2019-08-02 21441, 2019
ruaok
but if you go 32 days from day 1, you'll *always* be in the next month.
2019-08-02 21446, 2019
pristine__
You mean fixing 32?
2019-08-02 21424, 2019
pristine__
No matter what the RECOMMEDATION_GENERATION_WINDOW is, once I subtract it from current date, I always add 32 and move to first date of month and repeat the procedure?
2019-08-02 21405, 2019
yvanzo
alastairp: thanks, I was not looking at the right place, just found more annotations with control chars
2019-08-02 21412, 2019
alastairp
yvanzo: yeah, I had a hunch when I saw the value of the row with `{'_store': <already some xml}`, but it still took a bit to break out of the multiprocessing thread
2019-08-02 21431, 2019
yvanzo
I was looking at the range SELECT text FROM annotation WHERE text ~ '[\x00]' LIMIT 1;
I _think_ what is happening is that if a subprocess from pool.imap(indexer, ..) causes an exeption, it's caught in this exception handler. the next thing it does is puts STOP on the queue, which causes the solr indexer workers to shut down
2019-08-02 21425, 2019
alastairp
however, it seems that pool.imap continues running through all of the arguments, and keeps putting things on the queue
2019-08-02 21407, 2019
alastairp
I don't know enough about how exception handling works with multiprocessing to know if this is exactly the cause
2019-08-02 21410, 2019
alastairp
however it seems that a reasonable solution might be to have a try/except inside the indexer function to prevent uncaught exceptions propagating up to the main thread
2019-08-02 21414, 2019
Lotheric has quit
2019-08-02 21437, 2019
Lotheric joined the channel
2019-08-02 21407, 2019
alastairp
yeah, I just double-checked with some demo code, that's exactly what's happening. The exception is caught, but then pool.close()/pool.join() causes the process to wait until all items in the pool have finished processing
hi amCap1712 I’m really sorry because I couldn’t fully review the PR (too much work preparing the last items of the trip). Please, feel free to merge. But remember these last weeks that our goal is to leave the app fully functional, rather that styling or more features/support that can come later. Once last features are ready, test in several conditions: WiFi, cell data, offline, different screen sizes… Let’s
2019-08-02 21421, 2019
gr0uch0mars
keep the user experience small but amazing. Keep this level, you’re working very hard and it’s a pleasure to see the progress of the project. I’ll reach to you as soon as I’m there with an internet connection. Regards!
2019-08-02 21400, 2019
iliekcomputers
ruaok: PST now
2019-08-02 21436, 2019
alastairp
yvanzo: nice! in the end what did you do? handle the exception when adding to the queue?
