#metabrainz

amending unpushed commits is fine but don't do it for already pushed once :P

yeah, will keep in mind. i spotted it too late >_<

come to think of it I'm going to have to sign all my commits too ....

LordSputnik: could you purge @coveralls comments from https://git.io/vbbR1? I'm about to do a bulk signing commit that would leave only one coveralls comment for all my commits

then I'll do a proper writeup on the changes and submit

naiveai, by and large you shouldn't ever have to do a force push, so if you find yourself doing that, double-check.

Hey

I am trying to setup picard-website locally and following the install.md for linux

however when I am running pip install -r requirements.txt I am facing an error

Complete output from command python setup.py egg_info:

Traceback (most recent call last):

File "<string>", line 1, in <module>

File "/tmp/pip-build-g83s_q00/transifex-client/setup.py", line 14, in <module>

if long_description.startswith(BOM):

TypeError: startswith first arg must be str or a tuple of str, not bytes

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-g83s_q00/transifex-client/

How to solve this error?

Which version of python are you using?

Are you using docker?

I am using python 2.7.12

Using virualenv

*python 3.5.2

Leftmost: i have some…bad git habits to say the least, so I do force pushes more foten then I really should. i'm trying to shake off the habit

i feel like at this point I should just open a new PR without all the coveralls spam

lesson learned (i feel like i've been saying that a lot lately): git is literally magic, but magic should be used wisely.

LordSputnik, arthelon[m], Leftmost: made PR#174 with all commits signed, pushed all at once. submitted.

phew! now i won't bother you guys again.

@iliekcomputers any help?

zas: suffering some sort of server setbacks?

Stuff seems slow

Guest94268: I'm not familiar with Picard website but it uses python 2.7 I think

reosarevok: i did a quick check, it doesn't look slower than usual

I think now it is a bit better again - I was having issues with stuff taking so long to send it broke scripts :/

iliekcomputers: ping. see if we can appear in the same place at the same time today.

Here right now

ok, first off. I'm really sorry.

second, I remember what drama the freedb data dumps are.

they create one file on disk for every CD.

it chews up inodes and does an insane amount of file opening/closing.

the process is painfully slow and ideally one would do an in-memory stream decompress to not bog down the filesystem.

And us creating a file for each user is similar?

exactly.

as much as I love the fact that one user can quickly find their own data, it makes consuming data dumps much harder for everyone.

I see.

I think we should make it really simple then and not think too much.

Tbh, there are only ~1200 users in LB right now.

correct.

but this becomes sub-optimal at 10k+, which isn't far out.

Okay.

so, I am thinking that we should set some file size limit. say 10MB or maybe 100MB.

then we pump listens into txt files.

normal is to make one JSON document per line.

Text files which aren't username specific

correct.

And each Json file contains usernames?

maybe just monotonically increasing numbers.

*json doc

yes. I forget, does the JSON now contain usernames?

I'm not sure but I don't think it does.

that would make it easy for us, but a bit harder for users since they would need to grep for their user names.

my natural instinct is to make it so that we create an enclosing JSON doc with the username and then output one per line.

but catcat would make that tough. having to load 1M+ listens into ram to be able to parse it, its also not very smart.

so, we need to find a simple way to delineate listens by users.

we could do that in the index file.

user -> filename, offset

user -> filename, offset, size

so that to find the data for a given user, they need to extract one file, seek to offset and read size bytes.

and it makes dumping the data very simple.

Hmmm, sounds good, what if a user's listens take more than one file?

I think if we tread the filesize as a recommendation, not a hard limit, we should be good.

meaning that if we pick 10Mb as the limit (unlikely) and a user has 12mb of listens, they take up one file.

Ah okay

and if a user is at 9mb and then we reach catcat, fine.

catcat would still probably take many files?

I'm quite curious to dump catcat to a single file using your current setup.