#metabrainz

/

      • LordSputnik
        stanislas: it may be more efficient to set an attributes list

        • 2016-02-12 04303, 2016

      • LordSputnik
        Since this isn't user-facing we don't really need to worry about ease of use, and I seem to recall isinstance is relatively expensive

        • 2016-02-12 04349, 2016

      • reosarevok
        bitmap: huh. doesn't seem to work even then...

        • 2016-02-12 04353, 2016

      • stanislas
        Do you have some idea on how to do it without changing everyting ?

        • 2016-02-12 04334, 2016

      • LordSputnik
        stanislas: Well, for starters, you could just change get_attributes and the attribute definitions on the classes to put them in lists

        • 2016-02-12 04350, 2016

      • LordSputnik
        That's not a huge step, I think

        • 2016-02-12 04303, 2016

      • stanislas
        It is

        • 2016-02-12 04308, 2016

      • stanislas
        There are lots of them

        • 2016-02-12 04317, 2016

      • stanislas
        I could do this every __init__

        • 2016-02-12 04333, 2016

      • LordSputnik
        stanislas: you mean load the found attributes into a list?

        • 2016-02-12 04336, 2016

      • stanislas
        but this might be a little too slow

        • 2016-02-12 04343, 2016

      • stanislas
        yeah

        • 2016-02-12 04355, 2016

      • LordSputnik
        Hmm

        • 2016-02-12 04358, 2016

      • LordSputnik
        leave it for now

        • 2016-02-12 04308, 2016

      • LordSputnik
        We can improve it later if it's slowing things down :)

        • 2016-02-12 04326, 2016

      • reosarevok
        bitmap: not even that seems to work

        • 2016-02-12 04335, 2016

      • reosarevok
        Must be doing something wrong, but dunno what :/

        • 2016-02-12 04338, 2016

      • LordSputnik
        stanislas: Ok, and one other thing - why keep from_json and fetch_from_json separate?

        • 2016-02-12 04340, 2016

      • stanislas
        And it would look really bad after moving it all to list initlaizations in each class

        • 2016-02-12 04301, 2016

      • bitmap
        Huh

        • 2016-02-12 04309, 2016

      • stanislas
        from historic reasons :)

        • 2016-02-12 04324, 2016

      • stanislas
        When fetch_from_json was overloaded

        • 2016-02-12 04302, 2016

      • stanislas
        I will remove those functions in a while

        • 2016-02-12 04317, 2016

      • stanislas
        (fetch_from_json and fetch_from_json_filled

        • 2016-02-12 04302, 2016

      • LordSputnik
        stanislas: OK :)

        • 2016-02-12 04310, 2016

      • LordSputnik
        Now, what does Agent do?

        • 2016-02-12 04327, 2016

      • bitmap
        What if you explicitly set .css('color', 'white') in the code

        • 2016-02-12 04357, 2016

      • stanislas
        In future it is going to store all data related to connection, like host name, oauth things

        • 2016-02-12 04301, 2016

      • reosarevok
        ooooh.

        • 2016-02-12 04309, 2016

      • stanislas
        For now it is only host name

        • 2016-02-12 04332, 2016

      • LordSputnik
        OK, good :)

        • 2016-02-12 04350, 2016

      • LordSputnik
        And what are self.parse and self.cls for on the Attribute?

        • 2016-02-12 04350, 2016

      • stanislas
        parse is used for lists (like aliases)

        • 2016-02-12 04303, 2016

      • stanislas
        it is called before assigning value

        • 2016-02-12 04314, 2016

      • stanislas
        or dates

        • 2016-02-12 04321, 2016

      • stanislas
        to parse it to datetime format

        • 2016-02-12 04347, 2016

      • LordSputnik
        OK, so it transforms the input value to a value that can be processed by self.cls?

        • 2016-02-12 04316, 2016

      • stanislas
        There are no places when both cls and parse are used

        • 2016-02-12 04323, 2016

      • stanislas
        cls is for example for Alias

        • 2016-02-12 04338, 2016

      • stanislas
        so you can just write Attribute('alias', cls=Alias)

        • 2016-02-12 04340, 2016

      • LordSputnik
        stanislas: does anything bad happen if both are used?

        • 2016-02-12 04348, 2016

      • stanislas
        no

        • 2016-02-12 04309, 2016

      • LordSputnik
        OK :)

        • 2016-02-12 04314, 2016

      • LordSputnik
        Yea hthat makes sense

        • 2016-02-12 04332, 2016

      • stanislas
        First the value from JSON data will be parsed, and then provided to cls.from_json

        • 2016-02-12 04339, 2016

      • stanislas
        and then finally assigned

        • 2016-02-12 04354, 2016

      • stanislas
        It makes code a lot clearer

        • 2016-02-12 04308, 2016

      • reosarevok
        bitmap: ok, found *that* problem :p

        • 2016-02-12 04319, 2016

      • bitmap
        ?

        • 2016-02-12 04345, 2016

      • LordSputnik
        stanislas: what's happening when get_value_by_ws_name is given a tuple ws_name?

        • 2016-02-12 04347, 2016

      • stanislas
        In future I will probably change the name parse to parse_json, and add also parse_to_json

        • 2016-02-12 04350, 2016

      • reosarevok
        So, I was setting the class to menu-header, not the list

        • 2016-02-12 04301, 2016

      • reosarevok
        So of course > menu-header didn't find anything

        • 2016-02-12 04302, 2016

      • bitmap
        Doh

        • 2016-02-12 04356, 2016

      • stanislas
        see RelationshipEntity for example

        • 2016-02-12 04308, 2016

      • stanislas
        It's for nested dicts

        • 2016-02-12 04324, 2016

      • reosarevok isn't very smart sometimes :p

        • 2016-02-12 04325, 2016

      • reosarevok
        Sorry!

        • 2016-02-12 04326, 2016

      • stanislas
        it's in relationship.py

        • 2016-02-12 04331, 2016

      • stanislas
        That's clear ?

        • 2016-02-12 04359, 2016

      • LordSputnik
        stanislas: Ahhh, I think I see

        • 2016-02-12 04302, 2016

      • stanislas
        I can prepare a gist that show how it works

        • 2016-02-12 04309, 2016

      • LordSputnik
        So it's effectively moving down a path of properties?

        • 2016-02-12 04316, 2016

      • stanislas
        yeah

        • 2016-02-12 04336, 2016

      • LordSputnik
        eg. ('entity', 'relationship', 'relationship_id') would get you obj.entity.relationship.relationship_id ?

        • 2016-02-12 04356, 2016

      • LordSputnik
        it could be nicer to do that with recursion (eg. return get_value_by_ws_name(json_data[ws_name[0]], ws_name[1:]))

        • 2016-02-12 04310, 2016

      • LordSputnik
        but that's just a suggestion, both ways are equivalent

        • 2016-02-12 04331, 2016

      • LordSputnik
        OK, so happy with base.py

        • 2016-02-12 04337, 2016

      • stanislas
        :)

        • 2016-02-12 04300, 2016

      • stanislas
        yeah, it will get obj['entity'].['relationship.relationship

        • 2016-02-12 04323, 2016

      • stanislas
        obj['entity'].['relationship'].['relationship_id']

        • 2016-02-12 04327, 2016

      • LordSputnik
        It'd be good to use properties for eg. Creator.begin and end, so that they look like plain attributes

        • 2016-02-12 04348, 2016

      • LordSputnik
        https://docs.python.org/2/library/functions.html#…

        • 2016-02-12 04306, 2016

      • stanislas
        I remember having some reason for removing it but i can't remember it now

        • 2016-02-12 04315, 2016

      • stanislas
        * recall

        • 2016-02-12 04355, 2016

      • LordSputnik
        OK

        • 2016-02-12 04358, 2016

      • stanislas
        I will check it once again

        • 2016-02-12 04314, 2016

      • LordSputnik
        does the user have to manually call RequestQueue.send_all?

        • 2016-02-12 04329, 2016

      • LordSputnik
        and do they have to instantiate it?

        • 2016-02-12 04303, 2016

      • stanislas
        no

        • 2016-02-12 04305, 2016

      • LordSputnik
        (does it interface with the models yet?)

        • 2016-02-12 04311, 2016

      • stanislas
        they are not dealing with it at all

        • 2016-02-12 04354, 2016

      • stanislas
        And it doesn't iterfere with the models

        • 2016-02-12 04321, 2016

      • LordSputnik
        stanislas: oh no I meant do the models use it yet?

        • 2016-02-12 04327, 2016

      • LordSputnik
        I've just seen it on Entity()

        • 2016-02-12 04349, 2016

      • LordSputnik
        So a RequestQueue is instantiated for each group of requests to build an object

        • 2016-02-12 04350, 2016

      • stanislas
        they are :)

        • 2016-02-12 04307, 2016

      • LordSputnik
        And then all the requests for the object and it's properties to be filled are put into the queue, and then sent?

        • 2016-02-12 04308, 2016

      • stanislas
        And yes it is

        • 2016-02-12 04325, 2016

      • stanislas
        yes, but in two rounds sometimes

        • 2016-02-12 04359, 2016

      • stanislas
        in the first round: entity, relationships, aliases, identifiers ...

        • 2016-02-12 04316, 2016

      • stanislas
        in the second round (for Edition): publication, publisher

        • 2016-02-12 04344, 2016

      • LordSputnik
        OK, because it relies on having the edition data?

        • 2016-02-12 04349, 2016

      • stanislas
        \yes

        • 2016-02-12 04300, 2016

      • stanislas
        I don't know what publisher/publication to query

        • 2016-02-12 04315, 2016

      • stanislas
        before requesting Edition first

        • 2016-02-12 04323, 2016

      • stanislas
        ly

        • 2016-02-12 04333, 2016

      • LordSputnik
        Technically, it would be better to more everything to the second round apart from the data - the reason the WS serves URLs for that is so that the URLs can change if we need them to. However, so long as we update pybb if the URLs change, we should be OK

        • 2016-02-12 04356, 2016

      • stanislas
        good point, I will change it in some time

        • 2016-02-12 04359, 2016

      • LordSputnik
        OK :)

        • 2016-02-12 04307, 2016

      • LordSputnik
        Generally though, this is really good stuff :)

        • 2016-02-12 04333, 2016

      • stanislas
        I don't know if you've seen, but there is a function that can return a list of entities of various types.

        • 2016-02-12 04353, 2016

      • LordSputnik
        where?

        • 2016-02-12 04330, 2016

      • stanislas
        So if you want to query Works, Creators, Editions at once, it will return a list that will look like this [Creator, Creator, ..., Edition, Work, Edition]

        • 2016-02-12 04342, 2016

      • stanislas
        in entity.py

        • 2016-02-12 04300, 2016

      • stanislas
        get_multiple_ids_derived

        • 2016-02-12 04316, 2016

      • stanislas
        And thank you :) I am trying to do my best

        • 2016-02-12 04336, 2016

      • LordSputnik
        Ahh OK, I see now

        • 2016-02-12 04313, 2016

      • LordSputnik
        With a little bit of adjustment, you should be able to merge get_multiple_ids_json and get_multiple_ids_json_derived

        • 2016-02-12 04319, 2016

      • LordSputnik
        I think only 1 line is different

        • 2016-02-12 04331, 2016

      • stanislas
        Also, if you query multiple ids, requests from each round will be sended at the same time

        • 2016-02-12 04345, 2016

      • LordSputnik
        Also don't forget to update the licenses to v2 and also remove me from the copyright, since I haven't written any of it ;)

        • 2016-02-12 04359, 2016

      • stanislas
        That's not true

        • 2016-02-12 04304, 2016

      • stanislas
        you have written a lot of it

        • 2016-02-12 04311, 2016

      • stanislas
        I've been copying bbschema

        • 2016-02-12 04332, 2016

      • stanislas
        second, there is a TODO about moving to v2

        • 2016-02-12 04337, 2016

      • LordSputnik
        Nah that doesn't count, you've only really kept the class names

        • 2016-02-12 04325, 2016

      • LordSputnik
        (and maybe a few comments)

        • 2016-02-12 04329, 2016

      • stanislas
        that's also no true, see utils.format_date

        • 2016-02-12 04338, 2016

      • stanislas
        it's entirely yours

        • 2016-02-12 04309, 2016

      • LordSputnik
        well OK, there should be a header in that file with me on it, but I probably shouldn't be on the other files, trust me ;)

        • 2016-02-12 04321, 2016

      • stanislas
        Yeah, I see your point

        • 2016-02-12 04319, 2016

      • stanislas
        removed fetch_from_json and fetch_from_json_filled

        • 2016-02-12 04302, 2016

      • stanislas
        I have a question, what was it that you've said that I want to merge

        • 2016-02-12 04312, 2016

      • stanislas
        fixtures.py in bbschema ?

        • 2016-02-12 04316, 2016

      • stanislas
        something like that

        • 2016-02-12 04349, 2016

      • LordSputnik
        fixtures.py in bbws and utils/data.py in bbschema

        • 2016-02-12 04309, 2016

      • stanislas
        And how should I use this in bbws ?

        • 2016-02-12 04319, 2016

      • stanislas
        which database should I fill

        • 2016-02-12 04320, 2016

      • LordSputnik
        I'm not sure, it may not be worth doing, since it'll be ported to node soon

        • 2016-02-12 04335, 2016

      • stanislas
        ok, that's true

        • 2016-02-12 04353, 2016

      • stanislas
        How should I test delete/put/posts on pybb ?

        • 2016-02-12 04327, 2016

      • stanislas
        I wanted to set up my own bbws

        • 2016-02-12 04314, 2016

      • LordSputnik
        stanislas: yeah go for it :) The best thing would be to use data dumps to set up a db and then spin up the WS

        • 2016-02-12 04331, 2016

      • LordSputnik
        I really want to make Docker images for BookBrainz :P

        • 2016-02-12 04307, 2016

      • LordSputnik
        stanislas: I can make you some up-to-date production database dumps if you like?

        • 2016-02-12 04314, 2016

      • stanislas
        I've learnt what docker is no more than a week ago and it seems great :)

        • 2016-02-12 04335, 2016

      • stanislas
        I'd be grateful

        • 2016-02-12 04310, 2016

      • LordSputnik
        stanislas: ha, well I learnt about it about 2 days ago :P

        • 2016-02-12 04321, 2016

      • LordSputnik
        I sort of knew what it was but it seems amazing

        • 2016-02-12 04322, 2016

      • stanislas
        Do you have time to send me some data dump today ?

        • 2016-02-12 04346, 2016

      • LordSputnik
        stanislas: already made it :)

        • 2016-02-12 04300, 2016

      • LordSputnik
        just putting it somewhere public

        • 2016-02-12 04319, 2016

      • LordSputnik wonders why he hasn't set up weekly dumps yet

        • 2016-02-12 04307, 2016

      • stanislas
        because there are 500 books :) ?

        • 2016-02-12 04302, 2016

      • LordSputnik
        stanislas: But the database *does* change

        • 2016-02-12 04315, 2016

      • LordSputnik
        http://bookbrainz.org/bbdump-core-20160212.tar.bz2 http://bookbrainz.org/bbdump-derived-20160212.tar… http://bookbrainz.org/bbdump-editor-20160212.tar.…

        • 2016-02-12 04322, 2016

      • LordSputnik
        Import using bbschema/utils/import.py

        • 2016-02-12 04321, 2016

      • LordSputnik
        stanislas: first you'll need to create the database and blank tables though

        • 2016-02-12 04326, 2016

      • LordSputnik
        using utils/create.py