Gentlecat: But using classes made things easier to handle in the code.
alastairp: not sure what do u mean? How are you exactly importing the data?
Gentlecat: I have to constantly access the various values of tokens in the code (ex, token_id, api_key from which token was created, user_id who is attached to the token)
alastairp
armalcolite: to extend what Gentlecat is saying, in acousticbrainz and listenbrainz our policy has been to pass around objects as dictionaries
rather than creating an explicit class to carry the data
Gentlecat
yeah, I see what you mean
alastairp
it's true that classes help encapsulate data, but we haven't been doing that
Gentlecat
I've been actually thinking about this
there are some benefits, but it's easy to make it unnecessarily complex
alastairp
I suspect Gentlecat's comment is pointing out that we generally don't do this, and so adding a second way of working with data can make it complicated to understand what's going on
armalcolite
and, making the classes static reduced the number of calls that i have to make to achieve something
Gentlecat
it's easier to understand when you just see what comes in and what comes out
kind of functional-ish style
alastairp
although, to add to this, we didn't have have significant database access before armalcolite added his stuff (compared to e.g. acousticbrainz)
armalcolite
Though the logic in the db codes is pretty naive
alastairp
if we get to this stage, I wonder if going full-sqlalchemy ORM is a way to go
armalcolite
they have no authentication, so complexity is not much
Gentlecat
(I have CB to compare with, though it's deep in ORM hell)
alastairp
ah.
I guess you have counter examples there then :)
interesting
Gentlecat
it takes me a lot of time to understand some parts now that I'm coming back to it
alastairp
right
Gentlecat
nothing beats the good old SQL query
alastairp
I agree, I prefer to write and understand sql
though there are some ways of using the ORM which are almost a direct translation of the sql
Gentlecat
yeah, you have to be very careful though
armalcolite
\me agrees with Gentlecat comlpetely
alastairp
armalcolite: regarding your question about importing
I have a quick question which i'm hoping for some help on
alastairp
mellowdoubt: hi! ask away!
if it's specifically related to adding or editing data on the musicbrainz website you might have more luck in #musicbrainz
armalcolite
yes, i extract the epoch to compare on the show listens page.
alastairp
armalcolite: did you copy this query directly from the casandra code?
armalcolite
yes
mellowdoubt
if an artist uses an ampersand (&) symbol in there name as opposed to calling themselves x and x should that be changed on the database
armalcolite
This shows the listens in the range specified on the listens page.
mellowdoubt
my specific example is the band 'jessica and the fletchers'
alastairp
psycopg2 has some nice features where if you select a timestamp object, it will create a python datetime object with the correct timezone
armalcolite
yes, but the show listens logic needed to be changed then.
alastairp
so, just because the cassandra code selected unix timestamps, and converted them in python, it doesn't mean we should do the same in postgres
sure! that's part of the migration
mellowdoubt
on most releases they tend to use the '&' so should it be changed to this or should the existing name be left alone?
alastairp
mellowdoubt: reosarevok may be able to help you with that, but I think you'll get better help in the #musicbrainz channel. In here we are mostly talking about developement, rather than use of the database
sorry I didn't manage to review the code before we merged it, I may have noticed it earlier, but I think we should make this change
mellowdoubt
ah, sorry <alastairp> i'll try and switch channels :)
alastairp
no worries! many people are in both channels, so you'll get a response anyway
but perhaps the other one will get you one faster
mellowdoubt has left the channel
I'm thinking back to our original API specification. I see we use timestamps in query parameters and return values. I can't remember why we decided to do that instead of formatted dates
and if it was a design decision or a result of what cassandra forced us to do
armalcolite
actually, from what i remember
cassandra used partitions and you used this info to separate listens into partitions.
(i am not completely sure)
alastairp
yes, but internally I don't know if we stored the listen date just as a number (representing the timestamp) or as a date object with a semantic meaning
our tools let us treat listen dates as real objects, so I believe we should take advantage of this
armalcolite
we used to store only the timestamps everywhere.
hence, i needed to introduce 'extract(epoch from ts)' to mimic this feature.
while adding postgres
using timestamps in the query params is much better than using formatted dates?
alastairp
yeah, I understand. it was a reasonable choice to make when replicating the behaviour of cassandra
I don't have a strong opinion on either option
armalcolite
the problem with the current code is that fromtimestamp() returns localtime
alastairp
right
armalcolite
if we can just make that part to return utc() then our problem is solved.
alastairp
utcfromtimestamp doesn't :)
armalcolite
awesome!
alastairp
however, to me this still sounds like a solution to a problem that we shouldn't have in the first place
my preference would be for Listenstore to return datetime objects
we can convert these to timestamps when we need to display them
armalcolite
i thought of that solution earlier.
my idea was:
Using the built-in DB feature to extract the timestamp would be much more fast than iterating and than converting timestamp for all of them.
but i now think it is just a handful of listens and it wont make much difference.
alastairp
I don't understand what you're saying
don't consider it in terms of the computation time required to perform an action. no matter what we do we are not going to run into scalability problems with this example
diana_olhovyk_ has quit
instead we should use the tools which make our task easier for us
time is *hard*
it is very hard
armalcolite
I meant using extract(epoch from ts) would be faster than using datetime conversion to get timestamp.
alastairp
in fact, we've already shown that if we don't use the tools available to us, we cause problems in our code
armalcolite
sure! i'll keep this in mind.
alastairp
so please use the timestamps from postgres
the dates, I mean
armalcolite
ah, ok.
i'll do it now.
alastairp
the import of now listening songs is probably more important
please make a ticket for it too, so that we can follow what you're working on