"By sorting the files by name, two subsequent files have little different characters between them. Apparently this is very favourable for the compression efficiency."
2021-03-16 07533, 2021
alastairp
listen data might be more locally compressable if you put all stuff from the same artist/album next to each ohter (for example)
2021-03-16 07554, 2021
ruaok
which makes sense since a lot of compression algs use sliding windows. so if your data in your window is more compressable...
2021-03-16 07558, 2021
alastairp
yep
2021-03-16 07505, 2021
ruaok
but that wont work for listens.
2021-03-16 07531, 2021
ruaok
unless you sort the dump by MBID, then maaaaaybe
2021-03-16 07558, 2021
alastairp
right. I don't think the changing parts of listens are stable enough that it'll give us a significant decrease in size
2021-03-16 07520, 2021
alastairp
but anyway, could be interesting to try a few experiments - randomly add files, or sort by a few different fields
2021-03-16 07527, 2021
_lucifer
Mr_Monkey, updated the PR.
2021-03-16 07515, 2021
Mr_Monkey
Yep, saw that, thanks :)
2021-03-16 07535, 2021
Mr_Monkey
_lucifer: I think the endpoints for /follow and /unfollow should follow the same treatment, no?
2021-03-16 07554, 2021
_lucifer
yeah, makes sense to move those to the api as well.
with the position of the comma it kind of implies that CB only exists to review places
2021-03-16 07557, 2021
reosarevok
I guess it does :D
2021-03-16 07502, 2021
reosarevok
Will amend
2021-03-16 07536, 2021
reosarevok
"Places are the only entity that can be reviewed in CritiqueBrainz yet cannot be rated in MusicBrainz" ?
2021-03-16 07538, 2021
alastairp
maybe even remove "are the only entity". "Places can be reviewed in CritiqueBrainz but cannot be rated in MusicBrainz. All other entities that can be reviewd in CB can be rated in MB"
i mean ${COMPOSE_PROJECT_NAME}_listenbrainz instead of $COMPOSE_PROJECT_NAME_listenbrainz
2021-03-16 07543, 2021
nifemi joined the channel
2021-03-16 07548, 2021
sumedh has quit
2021-03-16 07526, 2021
RikkoM has quit
2021-03-16 07532, 2021
Nizar joined the channel
2021-03-16 07542, 2021
Nizar has quit
2021-03-16 07502, 2021
Nizar joined the channel
2021-03-16 07521, 2021
RikkoM joined the channel
2021-03-16 07521, 2021
nifemi
Hello everyone, my name is Precious Oluwanifemi a third year undergraduate of computer engineering in Obafemi Awolowo University. I have a good experience working with ReactJs,NextJs and Typescript. I am interested on working on the Music-Reviews and Pin-My Jam ideas for ListenBrainz during the period of gsoc
2021-03-16 07507, 2021
_lucifer
hi nifemi, welcome to the community!
2021-03-16 07549, 2021
nifemi
Thank you so much @_lucifer, looking forward to it
2021-03-16 07515, 2021
yvanzo
reosarevok: Maybe some other cols could be moved to SMALLINT but it's a bit short in time to check potential implications and to decide now. There will be other schema changes.
2021-03-16 07520, 2021
reosarevok shrugs
2021-03-16 07529, 2021
reosarevok
Cannot see any implications, but maybe I'm missing something :)
2021-03-16 07547, 2021
alastairp
_lucifer: good catch. perhaps we should also run shellcheck over this script
2021-03-16 07551, 2021
reosarevok
Anyway, I'll be back in a while and we can maybe finish the blog once bitmap is around :)
2021-03-16 07521, 2021
_lucifer
alastairp: yup, will do that.
2021-03-16 07504, 2021
_lucifer
also, we have two docker-compose.integration.yml files one in docker and second in docker/jenkins.
2021-03-16 07517, 2021
_lucifer
do we need both or can they be merged?
2021-03-16 07552, 2021
ruaok
zas: you about today?
2021-03-16 07551, 2021
alastairp
_lucifer: yeah, there's a trick about that. in local development we mount . into /code so that we can run tests without rebuilding, but this doesn't work in jenkins due to our docker-in-docker setup. so we need a separate file with no source volume configured
2021-03-16 07526, 2021
alastairp
however, we could try and improve this by using docker-compose overlay files instead, so we have just 1 file and add another config file in one case (either jenkins or locally, I'm not sure which one is better)
2021-03-16 07512, 2021
_lucifer
oh ok, i'll open a ticket for now and look into again later
iliekcomputers: ahh, I see. so the SSH keys in the server configs are just bog standard and the installation gives its the restrictions for what it can access and all that. good to know.
2021-03-16 07541, 2021
_lucifer
alastairp, adding `|| true` at the end of that statement seems to work
2021-03-16 07522, 2021
alastairp
_lucifer: ah, right. I think that's fine
2021-03-16 07531, 2021
alastairp
because `docker image rm` requires a non-empty input, right?
2021-03-16 07547, 2021
alastairp
and if the filter doesn't return anything then it fails to run?
the rationale is that say i only touched the dump code, i don't want to wait for other tests to run. it would be nice to have something like `./test.sh dumps`
2021-03-16 07530, 2021
c1e0 has quit
2021-03-16 07502, 2021
c1e0 joined the channel
2021-03-16 07559, 2021
c1e0_ joined the channel
2021-03-16 07528, 2021
c1e0 has quit
2021-03-16 07528, 2021
alastairp
_lucifer: interesting idea. Do you know that you can already specify a file/test class/pytest argument to the end of test.sh?
2021-03-16 07553, 2021
alastairp
so this would be a way of further grouping multiple files together into logical categories?
2021-03-16 07515, 2021
_lucifer
yes, i specify the file name usually but specifying the entire file path and multiple files is a bit cumbersome
2021-03-16 07531, 2021
_lucifer
i was thing using pytest marks
2021-03-16 07535, 2021
_lucifer
*thinking
2021-03-16 07549, 2021
c1e0_ has quit
2021-03-16 07506, 2021
_lucifer
like we specify the database tests in BU so that those are not run on python 3
2021-03-16 07508, 2021
alastairp
one consideration here is that it requires someone to keep the marks up to date
2021-03-16 07510, 2021
c1e0_ joined the channel
2021-03-16 07513, 2021
_lucifer
yeah
2021-03-16 07511, 2021
c1e0 joined the channel
2021-03-16 07534, 2021
alastairp
could we improve this by better folder organisation? (just thinking out loud)
2021-03-16 07558, 2021
alastairp
that is to say, I'm not opposed to the ideas of marks, and we could try and add a few
2021-03-16 07515, 2021
_lucifer
better organisation would be a step better than what we have now.
2021-03-16 07516, 2021
alastairp
perhaps just use the existing pytest -m xx functionality for now
2021-03-16 07558, 2021
alastairp
related to this, I also want to merge integration tests and unit tests into a single docker-compose file, and maybe the same pytest run
2021-03-16 07510, 2021
alastairp
in which case, marks might be a good idea to separate these anyway
