#metabrainz

So what's the 1 single command that sets up everything? A shell script I assume?

one advantage of interactive tools is that it could do validation before it writes the config file (e.g. check if a database url is valid, or a directory exists), but in my opinion, I'd prefer to do that test in the code itself

I see

they use a tool called ansible, which is designed for automated system administration

Do we use ansible at MetaBrainz?

anyway, let's not get distracted

Right :)

mm, which command are you talking about?

LB? or metabrainz setup

Both

LB is a shell script, develop.sh, you ran it yourself

metabrainz setup uses ansible

btw, double-check your setup instructions. you got the syntax of the pip install command wrong

forgot the ```-r``` flag

yep

```pip install -r requirements.txt```

that's better

(btw, running your instructions from scratch in a new checkout of your code is a great way of testing that everything works)

yep. Adding it to the to-do list

some comments about gen_tables.py:

I always set up a main guard in my files. because if you don't and you import the file for whatever reason, it's going to run that code

How do I setup that guard? 👀

And while we're at it, could you also touch upon writing libraries in Python?

I was really confused by the __init__.py stuff for module declaration, etc

(Check the lib folder for the same)

https://stackoverflow.com/questions/19578308/wh... has a small description of it

Great! That explains some of it

in short, "if you want to import a file in a folder, the folder needs to have __init__.py in it to tell python that it's a module"

it gets a lot more complicated than that, but that's the basics

Pratha-Fish: OK, I've run gen_tables.py. what's next?

Do I have to include anything in the __init__.py file? I've kept it empty for now. and that seems to do the trick

that's part of the "it gets more complicated"

in short, if you're always doing "from lib import x" or "import lib.x" then no, you don't have to include anything in it

Got it

alastairp: after running hte gen_tables.py:

Just run any required script.

e.g. ```python mapper.py```

what does mapper.py do? I see no comment at the top of the file describing its purpose (and nothing in the readme!)

I'll add it asap :)

It basically grabs a random set of MLHD files

Oh wait.

To generate the random set of files, you'll have to run another module

```python lib/gen_test_paths.py 100 'warehouse/samples/random_file_paths.txt'```

The above command generates a list of 100 random files from MLHD, upon which the mapper.py works (for now)

perfect, I have this file now

Did it run in the first attempt?

yes

Oh great

Pratha-Fish: great to see that gen_test_paths uses argparse.

not so great to see that mapper.py has interactive prompts :)

this comes back to the previous discussion

Started using arg parse since the last time you introduced me to it :)

Not proud of the prompts in mapper.py haha. It was just for my personal testing purposes. I'll replace it ASAP

imagine if I wanted to run this 5 times each with a different configuration option (e.g. once with 1000 items, once with 10000, not running the cache, maybe with a different set of input files)

yeah, and you already know how to use argparse!

🎉

the great thing about argparse, is that you can fill in the help= argument to add_argument, and suddenly you have documentation

with no additional effort

Oooo

I'll check that one out

Pratha-Fish: got this error when running mapper

You probably tried using cached data

I did

I was about to touch that one. The script can't use any cached data on the first attempt because it has to run and create the cache first before using it

If you run it without cache once, and then use cache from the next time, it will run

I'll add some documentation about it

so that prompt isn't "generate a cache", it's "use an existing cache"?

but the cache is automatically generated?

what is this cache of?

I'm reading the code now

btw, another error:

looks like you're missing an `os.makedirs("warehouse/mapper_outputs", exist_ok=True)`

The big idea is,

loading MLHD files, and their respective MB tables (even from parquet) takes ages. So if you just want to test the mapping process, it's better to use previously cleaned MLHD data and then run updated mapper functions on that data instead of creating that data everytime (which takes a few minutes at leaset)

as I mentioned - great idea to run your instructions on a fresh checkout of the code to ensure that everything works as expected. for now I'm just making the directory now

alastairp: Noted that one as well

right, got it

so this is definitely a testing thing, rather than something that will be needed for running on the full dataset?

yes exactly

I also have a full production script ready BTW

the one that we used to convert MLHD to zstd

The rec_track_checker.py script was used for the same

But the script is basically redundant now, since the original MLHD data has been deleted. and we won't need to run the script ever again

Looks like I hardcoded the path

Pratha-Fish: sure, but that's definitely a big piece of work that you did and so we should keep the script and document it

alastairp: yes. In fact it's gonna be the base for our final script :)

Also, it's not an interactive file afaik, and uses a config.json file for it's settings :)

good

remember though, that json doesn't allow for comments

so config.py would be much better, because 1) you can write comments in it, 2) you don't have to explicitly load the file and read data from it, you can just import it

Definitely. I'll switch it up to .py in the upcoming iteration

Do I also include the "main guard" in this config.py file?

only in files that you launch from the commandline

_what if someone runs config.py from the commandline_

I mean, no sane person would. Just curious lol

good question

it'll "execute" the file

however, what does the file include? 17 statements of the form "CONFIG_ITEM=1"

so this has no effect on your system

indeed

it's not going to load a file, delete a file, start a 3 hour computation, etc...

Yep

Which also makes me wonder if I should include this main guard in the modules

Because the module isn't supposed to be run on it's own. We just need to import functions from it ig

what do you think the behaviour of the module would be if you ran it?

if you had a main guard, what would you run in this case?

well none of them run anything TBH

these modules are just function/class declarations

right, so running the module isn't going to do anything

and you don't have anything to put inside the main anyway

okie

So what else should we discuss

Maybe I should mention the fact that as you might have noticed, most of the repo is pretty "bootstrappy". i.e. The scripts and notebooks help ME get answer, and test out new stuff.

I am kinda confused, what purpose this code might fulfill to it's end user? We could ponder upon that question and maybe re structure the complete repo accordingly

this is definitely an interesting project because the result of running it is a new dataset and then we never need the code again

the notebooks are great for the exploration, and they've helped you understand the code well. I see two possible options

one is that we just "archive" the notebooks. Put them in a folder, make sure that they run, and that there is a description at the top of the notebook explaining what it was for

the other option is that we start thinking about what our final output of the project is in addition to the new dataset

I'm thinking reports/blog posts, etc

do you think it would be possible to turn each notebook into a small post explaining the key things that we learned by running it? I see things like pandas/arrow/csv tests. We also did some zst tests, right? This would be great to turn into a few small posts with code examples and images

then we'd have a great set of guidelines for other people to learn from

That's exactly what I had in mind too :))

Which is why I've already cleaned and archived some test notebooks that demonstrate different benchmarks

excellent

keep this in mind with the timeline for your project too - it's OK to spend a week writing up these notebooks. this is just as valid as "coding". We don't want to rush this at the end of the project

btw, did you get notifications of the changed dates? Did we let you know?

I did recieve a copy of the mail you forwarded to mayhem regarding date extension

I also checked my GSoC dashboard, which is currently showing my project end date as 24th Oct

yes, that's what we set it to. good. just checking that you have that date in mind too

Yes, I did have that one in mind

alastairp: Given the remaining work, how long do you think the rest of the project should take?

Maybe even stuff that I should improve upon

well, work fills to expand the time available

*Maybe even let me know what I can improve upon since my work time schedule since early august

I think that your todo list that you mentioned earlier is a good start. As I said at the beginning of the week we should start writing the final scripts with the assumption that the mapping is working, while we keep debugging the mapping

the other things that we can add to the list are your final written deliverables (be that blog posts about your experiments, or just a final post describing the entire project), and also a possible task about looking at the release mbids

like we discussed earlier this week

Great

I'll recollect everything we discussed in the past few days, and update the To-Do list

that would be great, thanks

That would make it easier to set time bound goals and all

you could also write a rough timeline, of when you think you'll be able to complete each of these steps

Yes I'll try