zas: not urgent but you can remove musicbrainz-redis-store-beta from the redis stats whenever you get a chance, it's been merged into musicbrainz-redis-store
btw, if anyone was logged out of beta let me know, out of curiosity (my session was preserved but there might've been race conditions during the move)
yvanzo
bitmap: can I help you with beta or redis?
bitmap
yvanzo: I finished merging the beta redis store into the prod one, so we can deploy #1529 to beta now if you're up for that
iliekcomputers: so, I've gotten very little to no feedback on the dups. everyone seems to be fixated on listen counts, rather than doing some digging to find dups.
I think the listen counts in production are wrong too and I dont want to spend time fixing them.
iliekcomputers
The listen counts in production as in the user listen counts or the global listen count?
ruaok
I think the think to do is actually start a migration that could make its way to production.
both probably
iliekcomputers
The global listen count is definitely incorrect
I remember opening an issue about it.
ruaok
so, by getting a production ready setup, we can create a more realistic comparison, that people can then ignore.
but I wonder if "this is being ignored" is more like "I can't find problem, but I can't say for sure, so I won't say anything."
iliekcomputers
I think it's the second.
ruaok
likely.
so, starting a production ready conversion.... I suppose first step might be for you to review the import script.
then to do an actual migration -- which will be a bit tricky.
iliekcomputers
Yeah, that sounds good. Happy to review.
Would a doc detailing the steps be helpful?
ruaok
I would need to start a new exchange and connect it to the incoming stream.
yes!
I'll setup for the review and the doc after this chat.
once the next change starts receiving listens (and the queue will grow significantly) we will need to trigger a new dump.
and then once the dump is done, then it will take some 12-24 hours to prepare and import the dump.
then we can connect it live and in theory we should have a clean and consistent database.
I'm somewhat concerned about the number of listens growing, but I think we should be ok.
so, right now my migration code is in a separate repo.
iliekcomputers
Could we just keep writing the listens in the queue and import simultaneously?
ruaok
that is exactly what I want to do.
iliekcomputers
Writing the listens to timescale
I meant to say
ruaok
ah, no, not ideal.
the insert will be most performant when you have insert in sequential time order from oldest to newest.
iliekcomputers
Okay, that makes sense.
ruaok
having listens stream in will break that.
what I suppose can try is to stream the listens to timescale *until* the import. then stop it, let the queue grow, do the import, then catch up.
I *think* that wont impact the import so badly.
iliekcomputers
If the rabbitmq queue falls over, we'll just start the process over again with something else I guess. Sounds reasonable to me.
ruaok
given that the import code is going to run once, by me, I'm not too keen on importing the code into lb-server and tidying it up to our strict standards.
can you review two scripts on their own and mainly look for logical errors?
is the second script that does the actual importing.
iliekcomputers
Okay, will read through when I get the time.
ruaok
ok, great.
I'll work up a migration doc next.
iliekcomputers
Great, thanks!
ruaok
the importer is a bit tricky, since it uses threads to run 5 insertions at the same time.
tuned to make the import move at a reasonable speed.
the two most critical functions are check_for_duplicates and import_dump_file
iliekcomputers
Thanks, I'll take a look.
zas
yvanzo: we can stop search-server (old one) containers right?
yvanzo
zas: it is still available for mirrors using LUCENE search, not sure it is still used.
zas
we need to sort this out, because those are using quite a lot resources we may use for something else. What about stopping them for a while and see if anyone complains (if they do, we'll see what to do, but unlikely starting those again, rather pointing them at solr), I remember we had some hacks in preparation to ease the move, but they were never finished, and lead to a lot of complexity.
so I guess current instances aren't updated since a while
yvanzo
bitmap: I changed staticbrainz project on jenkins to remove previous temporary container, but the actual issue is that this container is not run and thus doesn't have a build/ directory anyway. Doesn’t seem to be in use currently anyway.
ishaanshah
iliekcomputers: Hi, please ping me when you get time
