17:50 PM
alastairp
what if, when we got a new model and needed to re-dump an old mbid, we just re-dump the entire file again
17:50 PM
that way, when we uncompress over the top of an existing dump, it'll just overwrite the old one
17:51 PM
actually, that's almost the same as saying "when we release a new model we make a new initial high-level dump"
17:51 PM
iliekcomputers
Yes. But how often are models released?
17:52 PM
alastairp
which I like - it removes a huge amount of complexity, and we don't plan to release new models very often
17:52 PM
I don't think we'd make new model releases more than 2 or 3 times per year
17:53 PM
it means we'd have to keep track of highlevel and lowlevel json dumps differently
17:53 PM
(currently we use just one row in `incremental_dumps` for everything)
17:53 PM
iliekcomputers
That wouldn't be hard to do i guess.
17:54 PM
This sounds like a good solution to me.
17:54 PM
alastairp
me too - it sounds familiar
17:55 PM
I think we came up with the same solution
17:56 PM
iliekcomputers
Lol :)
17:57 PM
alastairp: I'll open a few tickets and start working on them, I guess. We can supersede AB-97 with a new one saying no dump file should have more than X recordings.
17:57 PM
BrainzBot
17:57 PM
iliekcomputers
And one for the hl Json stuff
17:58 PM
alastairp
hold of on closing 97 for now
17:58 PM
iliekcomputers
Right, not closing it. Will just open a new one.
17:58 PM
alastairp
I'd like to see how big a 100k or 200k or 500k file is first
17:59 PM
because I don't think we should go less than 100k in a dump-part
17:59 PM
but if that's 5-10gb, then it's too big for a sample dump
17:59 PM
(having said that, we should be able to make a sample dump of x size or n number using the dump script)
18:00 PM
jsturgis joined the channel
18:00 PM
iliekcomputers
Right.
18:01 PM
A sample dump would be at most 1-2 GB, I guess.
18:03 PM
alastairp
I think that the absolute maximum we should have is 1gb
18:04 PM
maybe even less compressed - 2GB uncompressed hl and ll combined
18:04 PM
or something
18:05 PM
this is for people who might want to set up a dev environment to look at the server; or who want to play with the data for running experiments
18:05 PM
they don't need lots, just enough to make a basic dataset, etc
18:06 PM
iliekcomputers
I get the use case, just having set up AB too :P
18:17 PM
AB-314, AB-315
18:17 PM
BrainzBot
18:17 PM
18:34 PM
CatQuest
"D/T" it is then I guess
18:35 PM
ugh you know what is annoying?
18:36 PM
the "spam" measurments we have on the wiki not cikcing in until I'm half way trought typing a sentence
18:36 PM
Leftmost
If you say "Leftmost", that will be not-nice. :P
18:36 PM
iliekcom- has quit
18:36 PM
ferbncode has quit
18:37 PM
CatQuest
i am sure I'm more annoying thna leftmost anyway
18:37 PM
@I'm QuestMost
18:42 PM
alastairp
great, thanks iliekcomputers
19:02 PM
CatQuest
uhm, hey have you done anything to the stuff behind the tagging ability?
19:02 PM
i cant tag fro mthe sidebar anymore (the box in the "tags" tab still works)
19:03 PM
I could tag things just yesterday so..
19:08 PM
ListMyCDs: thanks for already finding this and making a bug ticket <3
19:14 PM
samj1912
19:14 PM
me and zas are trying to dockerize this
19:14 PM
zas: said you'd know about how to handle the pgsql plugins part
19:15 PM
bitmap
I thought it was already mostly dockerized
19:15 PM
samj1912
nope
19:15 PM
Is it?
19:16 PM
19:16 PM
bitmap
yes
19:16 PM
samj1912
but its missing components
19:16 PM
bitmap
what's missing?
19:16 PM
samj1912
mainly sir
19:16 PM
and the indexing components
19:17 PM
19:17 PM
hmm
19:17 PM
agentsim_ has quit
19:17 PM
bitmap
right, Gentlecat had already deployed this stuff using docker, so I assume something is working
19:17 PM
agentsim joined the channel
19:18 PM
samj1912
zas is not aware of the current config/parts and neither am I (docker noob here)
19:18 PM
bitmap
that shouldn't be hard to figure out
19:19 PM
samj1912
I have listed out all the parts in the manual setup, can you help us figure out if they are already dockerized
19:20 PM
zas: also pointed out a couple of mistakes with the existing config (data files in the same directory as config files)
19:20 PM
and also we might want to run a solr cloud setup rather than the normal solr
19:22 PM
bitmap
19:23 PM
samj1912
okay
19:23 PM
bitmap
those are meant to be in the same container
19:23 PM
samj1912
okay
19:25 PM
what about rabbitmq and connection to mbdb?
19:25 PM
bitmap
and afaiu mbdata and rngpy are just libraries used in the sir container
19:25 PM
samj1912
yes
19:25 PM
they are installed with sir
19:27 PM
bitmap
19:28 PM
19:28 PM
it's a config template, rendererd by consul-template (which pulls that above data from consul)
19:29 PM
samj1912
cool and where do we install the pg_amqp plugin in the main db?
19:29 PM
pointing to the above rabbitmq instance
19:29 PM
bitmap
that's already installed
19:30 PM
19:30 PM
but you'll need to insert a row into the amqp.broker table
19:30 PM
samj1912
okay, where will I have to do that?
19:30 PM
19:31 PM
bitmap
19:31 PM
has to be done manually currently (but it's just a one-time thing)
19:31 PM
samj1912
okay
19:32 PM
the host remains the same always?
19:32 PM
bitmap
yeah, it's where rabbitmq is running, which is always on serge right now
19:33 PM
there should already be a row in the broker table for the setup Gentlecat had, you can just use/modify that
19:33 PM
samj1912
okay
19:33 PM
I think that takes care of everything
19:34 PM
I will co-ordinate with zas to get a test setup via docker up and running
19:35 PM
thanks for the help :D
19:35 PM
bitmap
sure, let me know if you need help with the amqp setup
19:35 PM
samj1912
yup
19:36 PM
I haven't even looked into that part of SIR yet
19:36 PM
will have to understand how it works first
19:37 PM
bitmap
if you see queries like 'PERFORM amqp.publish(2, ...)' in sir, the '2' is actually the row id in the broker table
19:37 PM
that tells it where/how to connect to rabbitmq and where to publish things
19:37 PM
samj1912
okay
19:41 PM
bitmap: one last thing, how do I add the triggers generated by SIR to MB DB?
19:44 PM
bitmap
cat sql/CreateFunctions.sql | psql -U musicbrainz db_name
19:44 PM
cat sql/CreateTriggers.sql | psql -U musicbrainz db_name
19:45 PM
something like that :)
19:45 PM
samj1912
okay, so I just run in manually once?
19:45 PM
bitmap
(those files are in the sir repo)
19:45 PM
yes, unless the functions/triggers change and you need to update them
19:46 PM
samj1912
I haven't looked at the code yet, but I am not sure if the current triggers handle all the relationships and attributes I added over the last 2 weeks
19:50 PM
bitmap
you can regenerate them with the Makefile and see what changes
19:50 PM
`make createsql` I guess
19:52 PM
github joined the channel
19:52 PM
github
[acousticbrainz-server] alastair closed pull request #235: Upgrade javascript dependencies (master...js-upgrade)
https://git.io/v5ht6
19:52 PM
github has left the channel
19:53 PM
bitmap
samj1912: are you setting this up on the production database (bowie)? or somewhere else
19:53 PM
I mean if you're doing a test setup somewhere else first or something
19:54 PM
samj1912
I will first do it on a mbdb replicae
19:54 PM
19:54 PM
its a google cloud vm
19:55 PM
19:56 PM
bitmap
okay, not sure how postgres is setup there, you may need to install pg_amqp by hand
19:56 PM
samj1912
okay
19:57 PM
bitmap
19:57 PM
samj1912
yup, installed it on my local setup
19:58 PM
I have the entire thing running locally on solr-test
19:58 PM
I just need to shift it to docker
20:08 PM
bitmap: how do I call the GenerateSQLScript with correct arguments?
20:08 PM
20:08 PM
github joined the channel
20:08 PM
github
20:08 PM
github has left the channel
20:09 PM
samj1912
because I can see a lot of missing statements in the dropsql part
20:09 PM
I called it by `./GenerateSQLScripts.pl ../../sir/sql/ `
20:34 PM
agentsim_ joined the channel
20:38 PM
agentsim has quit
20:50 PM
20:50 PM
good night people o/
20:54 PM
aaa_ joined the channel
20:54 PM
aaa_ has quit
21:06 PM
CatQuest
natta samj1912 !
21:15 PM
Somasis is now known as squawk
21:15 PM
squawk is now known as Somasis
21:38 PM
bitmap
samj1912: it looks like it removed some stuff from the Drop* files that no longer exists, but nothing current is missing I don't think?
21:46 PM
kyan has quit
22:14 PM
upgrading jenkins to fix some security vulnerabilities
23:22 PM
MajorLurker joined the channel
23:46 PM
ferbncode_ has quit