I was working on making the scripts independent. I don't think we should still keep the htmls bidirectional.
2019-05-23 14330, 2019
pristine__
the HTMLs shall be independent, as the parent files.
2019-05-23 14357, 2019
pristine__
no?
2019-05-23 14314, 2019
ruaok
Is it more difficult to keep them bidirectional?
2019-05-23 14349, 2019
pristine__
umm....how will we pass HTML file names to scripts?
2019-05-23 14320, 2019
ruaok
Aren't we going to have job UUIDs? Use those for filenames and links
2019-05-23 14312, 2019
pristine__
for instance the first script, create_dataframes.py
2019-05-23 14348, 2019
pristine__
I do 'queries-%s.html.format(uuid4)'
2019-05-23 14338, 2019
pristine__
so how will the script train_models.py know this filename to pass it to html template?
2019-05-23 14357, 2019
pristine__
I think I am missing somethin?
2019-05-23 14302, 2019
Gazooo has quit
2019-05-23 14312, 2019
ruaok
Hmmm. I guess not. But then they can't be linked at all, no?
2019-05-23 14352, 2019
ruaok
It's not that important, go ahead without it if it is a problem.
2019-05-23 14351, 2019
Gazooo joined the channel
2019-05-23 14311, 2019
pristine__
ruaok: yeah. What I will do is, include necessary information in all html files so that the reader have all the info to understand that particular file.
2019-05-23 14336, 2019
pristine__
And should I tell you how the lookup time reduced?
2019-05-23 14346, 2019
ruaok
Please do!
2019-05-23 14301, 2019
ruaok is about to go AFK for the day
2019-05-23 14334, 2019
reosarevok
bitmap, yvanzo : does the ko issue GitHub is alerting about affect us?
2019-05-23 14300, 2019
pristine__
ruaok: we were using Rdds for lookup. To effectively use Rdds we must parallelize them which I still need to understand. Also, RDDs are slow. So we used dataframe for lookup (a dataframe by default has 200 partitions) and they are fast.
zas: I have to make deep changes to SEC workflow, did you set up anything I should preserve?
2019-05-23 14349, 2019
alastairp
and all functionality will be made free
2019-05-23 14316, 2019
zas
yvanzo: like ?
2019-05-23 14350, 2019
reosarevok
alastairp: but it can't automatically fix any bugs resulting from these dependency upgrades, right? Not sure what the usefulness is then :/
2019-05-23 14357, 2019
yvanzo
reosarevok, alastairp: in this specific case, we use a custom version of KO 2 with backported patches maintained by bitmap.
2019-05-23 14335, 2019
yvanzo
zas: I just noticed SEC has a unique workflow compared to other projects.
2019-05-23 14354, 2019
alastairp
reosarevok: when it was bought up the last time I commented that I didn't really like automatic tools for reasons similar to that. I was just copypasting from HN
2019-05-23 14358, 2019
zas
ah, yes, you can change it if needed
2019-05-23 14311, 2019
yvanzo
zas: For example, what distinguish Approved from Done?
2019-05-23 14331, 2019
alastairp
having said that, it makes a PR, rather than directly commiting the change, so tests can find bugs, and they can be fixed before merging
2019-05-23 14347, 2019
yvanzo
zas: Ok, I mostly need clearer statuses and direct transitions from Open to any.
2019-05-23 14324, 2019
yvanzo
alastairp: There is also an “Automated security fixes” option in GitHub.
2019-05-23 14338, 2019
yvanzo
That creates PRs too.
2019-05-23 14340, 2019
yvanzo
This is beta though.
2019-05-23 14310, 2019
Freso
alastairp: Related: https://dependabot.com/blog/gemnasium/ : "Finally, for us, Gemnasium's blog post is a warning of what can happen to businesses in a platform ecosystem. We believe Dependabot adds a lot of value over GitHub's dependency graph, and over Gemnasium, but if GitHub were to replicate our functionality they would likely crush us. We don't believe that's in their interest, but are staying as close to them as possible." :)
2019-05-23 14344, 2019
yvanzo
reosarevok, bitmap: I incidentally put mbs in beta of automated security fixes, PRs are disabled but I did not find how to remove that new tab :/
reosarevok: we have root/release/caa_darkened.tt but I guess it's not working for some reason (haven't looked)
2019-05-23 14309, 2019
bitmap
btw, if you have any technical details about those radio platforms not getting cover art, can you forward them to me?
2019-05-23 14332, 2019
aidanlw17
alastairp: I’m free to meet when you are
2019-05-23 14358, 2019
bitmap
reosarevok, yvanzo: I can backport the knockout patch so we can dismiss that alert
2019-05-23 14337, 2019
bitmap
I tried upgrading before and it broke too much stuff iirc
2019-05-23 14314, 2019
bitmap
so I'd rather focus on slowly removing it like jquery
2019-05-23 14326, 2019
yvanzo nods
2019-05-23 14319, 2019
alastairp
ferbncode: is there a test that would have picked up that problem on the homepage? if not, then it'd be good that spellew writes a test for that too
2019-05-23 14305, 2019
alastairp
it looks like there's no CI in CB too, if you need help setting that up then I'm sure someone could help you with it. iliekcomputers should be around next week
2019-05-23 14311, 2019
alastairp
aidanlw17: hi, how are you?
2019-05-23 14335, 2019
ferbncode
alastairp: right, I ran tests in my local setup and they all passed, there should be a test. I'll add it to the ticket.
2019-05-23 14351, 2019
ferbncode
sentry reports error in CB prod, and there is Jenkins that run tests, I'll ping iliekcomputers for deployment once it's fixed 👍
2019-05-23 14304, 2019
aidanlw17
alastairp: good! How are you?
2019-05-23 14341, 2019
alastairp
yeah, the issue with sentry of course is that it only notifies to admins once and will swallow subsequent occurrences of errors
2019-05-23 14301, 2019
alastairp
ferbncode: do you receive error messages from sentry for CB? If not, perhaps that might be something that we should enable
2019-05-23 14317, 2019
alastairp
aidanlw17: good, but the week's not finished yet
2019-05-23 14325, 2019
alastairp
did you try and import the data dump?
2019-05-23 14356, 2019
ferbncode
alastairp: I receive error messages for CB from sentry.
2019-05-23 14356, 2019
ferbncode
I'll be more proactive and keep an eye on them 😅
2019-05-23 14324, 2019
alastairp
ferbncode: we have integration in sentry to create tickets on jira from the sentry page
2019-05-23 14337, 2019
aidanlw17
alastairp: I did try and import the data dump, but I haven't been successful.. I've used `./develop.sh run --rm webserver python2 manage.py import_data path_to_the_archive` and `./develop.sh run --rm webserver python2 manage.py init_db --force path_to_the_archive` before to import smaller archives that I have made, but those were .tar.xz files and the .sql.bz2 doesn't work with that, is that correct?
2019-05-23 14305, 2019
aidanlw17
After trying that, I tried to copy the dump into the docker postgres container volume and use pg_restore
2019-05-23 14336, 2019
alastairp
ah, sorry that I didn't tell you how to import it
2019-05-23 14312, 2019
alastairp
right, so. import_data isn't correct, it's a postgres database dump
2019-05-23 14327, 2019
alastairp
but pg_restore is only for binary dumps, and this is a text dump
2019-05-23 14356, 2019
alastairp
so you should be able to run `bzcat [thefile] | psql`
aidanlw17: I found something else while reading last week too: from https://github.com/spotify/annoy - "another feature that really sets Annoy apart: it has the ability to use static files as indexes. In particular, this means you can share index across processes."
2019-05-23 14343, 2019
alastairp
this is interesting. it means that the lack of ability to update the index is a _feature_ of annoy. it's a tradeoff that they made to allow multiple processes to read the same index file
2019-05-23 14342, 2019
alastairp
so we should definitely see if we also need this tradeoff - either we want to do this, or it might be more important for us to update easily. I still think that it's OK that we start with annoy, but if we see our requirements change we might want to look at this again
2019-05-23 14346, 2019
bitmap
yvanzo: I guess there's no "In Review" step for SEC?
2019-05-23 14300, 2019
bitmap
also not sure what the difference between Resolve and Close is
2019-05-23 14359, 2019
alastairp
if it's the same workflow as AB, resolve turns it into "patch sent", and close turns it into "fixed"
2019-05-23 14325, 2019
alastairp
(or closed for any other reason, etc)
2019-05-23 14333, 2019
bitmap
hmm. here they both open the same screen and let you pick the resolution
2019-05-23 14336, 2019
aidanlw17
alastairp: interesting, I had previously read about its use of static files but I didn't make the connection to the update functionality
2019-05-23 14348, 2019
alastairp
bitmap: yes, but the state after you click ok will be different
2019-05-23 14349, 2019
yvanzo
bitmap: just update alert status on GitHub
2019-05-23 14351, 2019
yvanzo
bitmap: we should probably have an MBS ticket for that
2019-05-23 14356, 2019
aidanlw17
alastairp: So it could be helpful for us in terms of using the index in multiple ways at the same time, but if our update mechanism doesn't work we could look for something that updates rather than allowing multiple processes?
2019-05-23 14322, 2019
alastairp
right. I'm still a little unclear about what our final result of this analysis will be
2019-05-23 14351, 2019
alastairp
for example, if the annoy index is small enough, perhaps we could just distribute this to people who want to be able to compute their own similarity?
2019-05-23 14307, 2019
alastairp
but I have no idea what the size will be until we make it
2019-05-23 14321, 2019
aidanlw17
Yeah that is unclear
2019-05-23 14341, 2019
alastairp
something that allows multiple access would be great if we wanted to make an API endpoint that uses it
2019-05-23 14359, 2019
aidanlw17
Do you mean make the index distributable rather than creating the api endpoints that use it?
2019-05-23 14304, 2019
alastairp
but if a separate tool returns responses fast enough, perhaps it doesn't matter if we can only make one query at a time
2019-05-23 14306, 2019
alastairp
right, exactly
2019-05-23 14337, 2019
aidanlw17
Yeah I see
2019-05-23 14345, 2019
alastairp
we currently have two possible use-cases that I'm aware of: "what are some similar items to mbid x" and "what are some similar items to [this lowlevel file]"
