November 2010 - cl-rdbms-devel@lists.common-lisp.net

[cl-rdbms-devel] hu.dwim.rdbms.oracle-timestamp-with-timezone-local-time-bug
by Marshall McLuhan 26 Nov '10

26 Nov '10

How do I handle such thing most efficiently? Should I report them at all? Did change the following in hu.dwim.rdbms/source/oracle/conversion.lisp (dunno if this does more harm than good without unit tests but atleast the error is gone) (def function make-timezone (hours minutes) (let ((offset-in-sec (* (+ (* 60 hours) minutes) 60))) (if (and (= minutes 0) (= hours 0)) +utc-zone+ (local-time::make-timezone :subzones `((,offset-in-sec nil "anonymous" nil nil)) :name "anonymous" :loaded t)))) to (def function make-timezone-old (hours minutes) (let ((offset-in-sec (* (+ (* 60 hours) minutes) 60))) (if (and (= minutes 0) (= hours 0)) +utc-zone+ (local-time::make-timezone :subzones (make-array 1 :initial-contents (list (local-time::make-subzone :offset offset-in-sec))) :name "anonymous" :loaded t)))) to get rid of: SYMORA> (with-symdb (execute [select * symtag])) --TIME MARK 2010-11-04-- 21:34 0 RDBMS DEBUG About to BEGIN transaction in database #<ORACLE {100DD2F9B1}> 21:34 0 RDBMS DEBUG Executing "SELECT * FROM symtag" STYLE-WARNING: Undefined alien: "OCIGetInstantClientPackageName" 21:34 0 RDBMS DEBUG Connecting in transaction #<#<STANDARD-CLASS NIL {10033B8C31}> :begin-executed-p #t {1002DFA061}> 21:34 0 RDBMS DEBUG Logging on in transaction #<#<STANDARD-CLASS NIL {10033B8C31}> :begin-executed-p #t {1002DFA061}> 21:34 0 RDBMS DEBUG Preparing command: "SELECT * FROM symtag" 21:34 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 21:34 0 RDBMS DEBUG Fetched: "foo" 21:34 0 RDBMS DEBUG Fetching "SYMTAG_CREATION_DATE" from buffer at index 0 21:34 0 RDBMS DEBUG Fetched: :NULL 21:34 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 21:34 0 RDBMS DEBUG Fetched: "foobar" 21:34 0 RDBMS DEBUG Fetching "SYMTAG_CREATION_DATE" from buffer at index 0 21:34 0 RDBMS DEBUG Fetched: :NULL 21:34 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 21:34 0 RDBMS DEBUG Fetched: "foo11" 21:34 0 RDBMS DEBUG Fetching "SYMTAG_CREATION_DATE" from buffer at index 0 --- *sbcl debug buffer* The value ((3600 NIL "anonymous" NIL NIL)) is not of type (SIMPLE-ARRAY T (*)). [Condition of type TYPE-ERROR] Restarts: 0: [TERMINATE-TRANSACTION] return (values) from the WITH-TRANSACTION block executing the current terminal action :COMMIT 1: [COMMIT-TRANSACTION] mark transaction for commit only and return (values) from the WITH-TRANSACTION block 2: [ROLLBACK-TRANSACTION] mark transaction for rollback only and return (values) from the WITH-TRANSACTION block 3: [RESTART-TRANSACTION] rollback the transaction by unwinding the stack and restart the WITH-TRANSACTION block in a new database transaction 4: [RETRY] Retry SLIME REPL evaluation request. 5: [*ABORT] Return to SLIME's top level. --more-- Backtrace: 0: (LOCAL-TIME::MAKE-TIMEZONE)[:EXTERNAL] Locals: SB-DEBUG::ARG-0 = 6 1: (HU.DWIM.RDBMS.ORACLE::LOCAL-TIME-FROM-TIMESTAMP-TZ #.(SB-SYS:INT-SAP #X006918C0) #<unavailable argument>) Locals: SB-DEBUG::ARG-0 = #.(SB-SYS:INT-SAP #X006918C0) SB-DEBUG::ARG-1 = :<NOT-AVAILABLE> 2: (HU.DWIM.RDBMS.ORACLE::FETCH-COLUMN-VALUE #<HU.DWIM.RDBMS.ORACLE::COLUMN-DESCRIPTOR {100B9E6761}> 0) Locals: SB-DEBUG::ARG-0 = #<HU.DWIM.RDBMS.ORACLE::COLUMN-DESCRIPTOR {100B9E6761}> SB-DEBUG::ARG-1 = 0 3: (CURRENT-ROW #<HU.DWIM.RDBMS.ORACLE::ORACLE-SEQUENTIAL-ACCESS-CURSOR {100B9E6E71}>)[:EXTERNAL] 4: (FOR-EACH-ROW #<CLOSURE (LAMBDA #) {100B9E8979}> #<HU.DWIM.RDBMS.ORACLE::ORACLE-SEQUENTIAL-ACCESS-CURSOR {100B9E6E71}>)[:EXTERNAL] 5: (COLLECT-ROWS #<HU.DWIM.RDBMS.ORACLE::ORACLE-SEQUENTIAL-ACCESS-CURSOR {100B9E6E71}>)[:EXTERNAL] 6: (HU.DWIM.RDBMS.ORACLE::EXECUTE-PREPARED-STATEMENT ..)[:EXTERNAL] 7: ((SB-PCL::FAST-METHOD HU.DWIM.RDBMS::EXECUTE-COMMAND (ORACLE HU.DWIM.RDBMS.ORACLE::ORACLE-TRANSACTION STRING)) ..)[:EXTERNAL] 8: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1. SB-PCL::.ARG2. SB-INT:&MORE SB-PCL::.DFUN-MORE-CONTEXT. SB-PCL::.DFUN-MORE-COUNT.)) ..) 9: (HU.DWIM.RDBMS::CALL-WITH-TRANSACTION* #<FUNCTION (LABELS HU.DWIM.RDBMS::WITH-TRANSACTION*-BODY) {10034E8619}>)[:EXTERNAL] 10: ((LAMBDA ())) 11: (SB-INT:SIMPLE-EVAL-IN-LEXENV (WITH-SYMDB (EXECUTE (SQL #))) #<NULL-LEXENV>) --more-- !DSPAM:4cd3227b48581611617599!

2 1

[cl-rdbms-devel] Cvenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 08 Nov '10

08 Nov '10

>> Maybe you don't expect an outsider to understand the "mechanics" of the code >> by reading your long names and >> merely want to aid that process of understanding. How well that is >> accomplished through the use of long names, is, of course, >> still open for question. :) > > >well, it certainly helps the process of understanding to have one less >indirection (e.g. not having to make the extra step to get from the >abbreviated short name to its original long form). > Agreed. Nonetheless, in an (my) ideal world you would be able to derive the long form from the abbreviated form by: 1. knowing the outlines of your territory (files, dirs, packages, systems, context of all kinds) 2. looking at and hopefully understanding the "mechanics" of the code Admittedly that isn't cheap, although it seems doable for an outsider. (eventually we are all outsiders) All in the name of circumventing names that are hard to understand. (for outsiders) The more I think about it I come to the conclusion that excessive use of long names are a way of "getting more intimate" with your code. That helps you personally (or a small group) over the short-, midterm (maybe even long-term or the rest of your life) but harms others. In the end, it seems to me, it all comes down to not having any standards and a practically infinite number of possible meanings. (costs of sharing meaning, transaction costs) >and don't forget that the frequency the name is "used" plays a very >important role regarding the decision about long/short names. you can >easily convince me about short names of frequently used constructs >(bind, let, ...), because even we prefer short names for them. but >these names are only a tiny portion of all the names introduced in the >codebase... > That makes sense to me. Sounds like you are making a case for using long names as an asset by having the contrast between long and short. And the costs of sharing meaning about a name that gets used frequently are easier to justify. Again, opens the question as to how much long names are capable of reducing costs of sharing meaning. Could they even increase the costs? I certainly made that case. Seems a little crazy from this perspective. Surely the more words you use to describe something the easier it gets to share what it actually means? Does purpose come into play now? Maybe I should stop here. > >the extremum on this scale, namely writing code with abbreviated short >names of infrequently used constructs is basically what code >obfuscater algorithms do. > Right, only context is missing in this picture. Hm, the costs of sharing meaning often seem to be exceptionally low when it comes to context. > >random thought re names: once a flat-text codebase has been parsed >into a graph, then all the human given names could be dropped because >they are redundant. the machine can execute the semantics encoded the >same way without identities having human parsable character strings. > >conclusion: names should/could have much less important role regarding >identity management and restoring a graph from flat-text, and much >more important role in helping human understanding of such a graph. >e.g. having a long and a short names for the same identity; the >ability to have personalized names (e.g. let shown as a special >graphical element in my editor instead of using a three letter word); >etc... > Absolutely > >i have a lot more thought on this (partly because we worked at >intentsoft.com) but it's a bit out of scope on this list... and > Impressive, a testimony from the New York Times. Yes, this is awfully off topic. As a side note: The relative low level interface of hu.dwim.rdbms I was talking about, on top of which as many extensions as possible can be build, seems to be already there. It's just that it isn't properly exposed, in my eyes at least. > >writing code has much more utility than talking about writing code... >:) > Hard to tell. On average you are probably right. > >but reading your mails makes me think that you are already getting far >ahead of most people when it comes to giving good names to >abstractions. spending brain cycles to think about the problem is >already more than what most people do. > I take this as an compliment. (normally I refuse compliments, tells me something) Regards, chris > >-- > attila > > !DSPAM:4cd81d6e48581330461277!

1 0

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 08 Nov '10

08 Nov '10

After all, you derived the long names from the "mechanics" of the code. And now you expect an outsider to go the opposite direction, deriving the "mechanics" of the code from the long names? I don't think this is going to work for an outsider. Long names may even confuse him because he has to "see through" them in order to be able to understand the "mechanics". Maybe you don't expect an outsider to understand the "mechanics" of the code by reading your long names and merely want to aid that process of understanding. How well that is accomplished through the use of long names, is, of course, still open for question. :) Regards, chris !DSPAM:4cd7f39b48584559617140!

2 1

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 08 Nov '10

08 Nov '10

One more critic. Constantly you have to come up with names that describe things that can't be (easily) described objectively and therefore make hardly more sense to an outsider than short names. I see unit tests as more import than long names (and shorter because of the use of short names) even if they seem redundant at times. They make a great documenation. A documentation that can test itself and describes code most accurately. In my opinion, the letters you save with short names should go into unit tests. That said, short names have their drawbacks, of course, and it's also a personal question. (who you came from, genetics etc) Regards, chris !DSPAM:4cd7ea3648581477279777!

1 0

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 05 Nov '10

05 Nov '10

To me, in the end, it is more like an art than a science. ( I guess this can be generally agreed upon ) I, like you, I suspect, see the need for complexity to be saved, like a precious resource, so that it is not being used up before it is really needed. For me this also includes starting with short names / using short names where possible. (UIGNI - You ain't gonna need it) But that doesn't mean that I'm not willing to use long names where I see fit. The opposite is more likely, I see the possibility of using long names in contrast to short names as an asset. It's just that I want to keep things as simple as possible for as long as possible, so that it can grow gracefully. (KISS - Keep It Simple Stupid) Nonetheless, I'm still willing to ignore my feelings in this regard in order to be able to work with other people and their sense of order and complexity. Although, I have to admit, it is kind of a neat idea (albeit not a necessity) to be able to follow my own conventions in the packages/systems I'm responsible for except for the public interfaces, these would follow traditional conventions. That, of course, would only work when starting from scratch. When editing others people work the old style has to be preserved, that, even I'm able too see :) Whatever ... Regards, chris !DSPAM:4cd456de48582047418725!

1 0

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 05 Nov '10

05 Nov '10

>>>we stick to long names (and use fuzzy completion in slime). >> >> I will try this fuzzy completion thing but I'm sceptical. Shorter names may >> still give you an advantage when >> it comes to reading code. :) > > >i suggest a bit more meditation on this. and if you still haven't >changed your mind, then run experiments that include returning to code >you wrote several months/years ago... :) Writing code in such a way that you can come back to it after years and understand it is certainly a skill. A skill I don't have much experience with. I will encounter this challenge soon enough, though. > >sometimes it happens that i accidentally implement something twice >because of patch organization problems and/or not having an unlimited >memory. to my delight, i more often than not name stuff exactly the >same way in both implementations which are quite distant in time. > As for patching I don't have much experience either. I have never really collaborated with others when writing code. I'm using bazaar tentatively to sync my stuff between desktop and laptop but that's about it. (well, and since recently some kind of production server) >it also means that when i happen to look for something, i'll grep for >at least a part of the name i would give it. this strategy usually >very cheaply gets me to what i'm looking for. even years later... > Global grep certainly benefits from long descriptive names but it can, as I see it, lead to weak files/directories and systems/packages structures. A grep of a sub-directory should do the trick even for short names. (short names may also appear in conjunction with package names) Ensuring that meaning is distributed equally on all levels (text/code, files/directories, systems/packages) should make short names more practical. I have to admit that I enjoy working with packages a lot, even as to go so far as to shadowing symbols so that I can reuse them as often as I see fit. Fictional example: (defpackage "CURL" (:export "GET")) (defpackage "FOO" (:use "CURL") (:shadow "GET")) (in-package "FOO") (defun get (arg) (do-additional-stuff (curl:get arg))) I'm also fond of using :shadowing-import-from. For me calling the same thing (or almost the same) with the same words is just to relieving. >all in all, having long and descriptive names is not a question for me >anymore, especially with slime's fuzzy completion. > I'm afraid I have to learn this the hard way even if it is just for finding the right balance between short and long. Writing down what some abbreviations stand for is also something I'm willing to do. (readme file, documentation string, comments) Also, in my opinion, the more local a name the shorter it can be. Enough of my strange customs. :) Wish you a nice weekend. ( P.S. So now you have it, hopefully not a debate. Let's make sure this stays dialogue. (and just in case you have doubts, I'm still willing to adapt)) > >> Just want to add that I'm grateful for your software, thank you. > > >we're glad you find it useful! > >-- > attila > > !DSPAM:4cd42ec048584894726250!

1 0

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 05 Nov '10

05 Nov '10

>we stick to long names (and use fuzzy completion in slime). I will try this fuzzy completion thing but I'm sceptical. Shorter names may still give you an advantage when it comes to reading code. :) You may argue that the win in clarity is well worth it. I would would argue that while it is true that short names can easily obfuscate code it's not necessarily so. But I don't want to get into this debate. When working with other people I would always adapt. Just want to add that I'm grateful for your software, thank you. Regards, chris !DSPAM:4cd4180e48581103713428!

2 1

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 05 Nov '10

05 Nov '10

>> While the sql reader syntax e.g. [select * table] is convenient for humans >> it doesn't allow for being used by other functions or does it? > > >(read-from-string "") as a last resort, otherwise it reads into a CLOS >based AST anyways and that you can construct programmatically. > >note: the SQL reader and AST stuff is on the TODO to be moved into >hu.dwim.quasi-quote and rebased on its infrastructure. but it's not >high priority, and wouldn't change anything conceptually... > > >> In my opinion such a sql reader syntax is nice to have but not being able to >> extend the software >> through the use of functions is disastrous in my eyes. I have the feeling >> that we create two camps here, one the developers >> and the other the users. Personally I don't like that distinction and would > > >it's a library for developers, there are no two camps. it's only developers.... > > I made a rash judgement here, sorry. >> much prefer to make the possibility of extending >> the software through the use of functions as easy as possible for the so >> called "user". >> >> With that in mind I find myself using internal function and classes from the >> hu.dwim.rdbms package >> ( e.g. sql-and, sql-sequence-nextval-column, sql-identifier, >> sql-all-columns, sql-= ) >> and writing wrapper functions for it e.g. > > >we didn't put too much effort into deciding what should be exported >and be part of a long-time API, and what isn't. but while (not) doing >that we were on the conservative side... > Maybe we should separate the sql reader syntax "stuff" from the rest, so we have one more package to maintain? :) Please make sure that you expose a "relative" low level interface that can be accessed programmatically and is as expressive as necessary so that a great variety of extension can be build on top of it. I find it somewhat puzzling that the sql reader syntax was not build on top of such an interface. After all, the sql reader syntax should be just an extension? If I think about I often work in one package despite the fact that two would be warranted just to get things done. Maintaining packages is hassle. So maybe not so puzzling after all ... > >> What do you think about what I just said. Did I go wrong somewhere? >> And if the above is really an issue, any ideas as how to handle it? > > >define your own convenience functions? if they seem to be generally >useful, propose an extension (but to spare some time, the naming >convention you use in your functions will not go though :) > Yes, I will stick to my convenience functions and adapt to your future changes. As for the naming convention I'm more than willing to compromise and there are many ways for dealing with this e.g. emacs abbrev mode, graham's abbrev macro or just inline functions for personal use. Don't worry, I never expected my naming convention to go through. > >> Right now I did create another system and package called >> hu.dwim.rdbms.oracle.ext for things like the above. >> (The names above are not carved into stone tablets. I often change my mind >> before I settle for certain names and yes, >> often the wrappers seem unnecessary but I don't want to change the package >> definition of hu.dwim.rdbms >> and/or prefer shorter names.) > > >that's a good compromise and leaves you the freedom to use >names/shortcuts you prefer. even if such utils were provided by the >main lib, it would not please everyone... so the way to go is to >provide a preferably complete skeleton of functionality using >descriptive and sometimes long names, and let people do their >shortcuts if the need be. > >we stick to long names (and use fuzzy completion in slime). > >-- > attila > > -- chris !DSPAM:4cd40a5848581824115346!

1 0

[cl-rdbms-devel] Convenience over extensibility? Or: How to build on top of hu.dwim.rdbms?
by Marshall McLuhan 05 Nov '10

05 Nov '10

Hi While the sql reader syntax e.g. [select * table] is convenient for humans it doesn't allow for being used by other functions or does it? In my opinion such a sql reader syntax is nice to have but not being able to extend the software through the use of functions is disastrous in my eyes. I have the feeling that we create two camps here, one the developers and the other the users. Personally I don't like that distinction and would much prefer to make the possibility of extending the software through the use of functions as easy as possible for the so called "user". With that in mind I find myself using internal function and classes from the hu.dwim.rdbms package ( e.g. sql-and, sql-sequence-nextval-column, sql-identifier, sql-all-columns, sql-= ) and writing wrapper functions for it e.g. (defun sqlid (name) (hu.dwim.rdbms::sql-identifier :name (string-upcase name))) (defun sqleq (left right) (hu.dwim.rdbms::sql-= left right)) (defun sqlallcol () (lst (hu.dwim.rdbms::sql-all-columns))) (defun sqlselect-single (tbl keycol-id keycol-val) (select (sqlallcol) (lst tbl) :where (sqleq (sqlid keycol-id) keycol-val))) (defun seqnext-literal (name) (make-instance 'hu.dwim.rdbms::sql-sequence-nextval-column :name (string-upcase name))) What do you think about what I just said. Did I go wrong somewhere? And if the above is really an issue, any ideas as how to handle it? Right now I did create another system and package called hu.dwim.rdbms.oracle.ext for things like the above. (The names above are not carved into stone tablets. I often change my mind before I settle for certain names and yes, often the wrappers seem unnecessary but I don't want to change the package definition of hu.dwim.rdbms and/or prefer shorter names.) Regards, chris !DSPAM:4cd3f8b248581688513886!

2 1

[cl-rdbms-devel] hu.dwim.rdbms.oracle-utf-problems
by Marshall McLuhan 04 Nov '10

04 Nov '10

Solved. Thank you -------- chris. * (select-symtag) --TIME MARK 2010-11-04-- 17:46 0 RDBMS DEBUG About to BEGIN transaction in database #<ORACLE {1003212741}> 17:46 0 RDBMS DEBUG Executing "SELECT SYMTAG_KEY FROM symtag" STYLE-WARNING: Undefined alien: "OCIGetInstantClientPackageName" 17:46 0 RDBMS DEBUG Connecting in transaction #<#<STANDARD-CLASS NIL {1005497791}> :begin-executed-p #t {100501B101}> 17:46 0 RDBMS DEBUG Logging on in transaction #<#<STANDARD-CLASS NIL {1005497791}> :begin-executed-p #t {100501B101}> 17:46 0 RDBMS DEBUG Preparing command: "SELECT SYMTAG_KEY FROM symtag" 17:46 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 17:46 0 RDBMS DEBUG Fetched: "foo" 17:46 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 17:46 0 RDBMS DEBUG Fetched: "foo11" 17:46 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 17:46 0 RDBMS DEBUG Fetched: "foobar" 17:46 0 RDBMS DEBUG Fetching "SYMTAG_KEY" from buffer at index 0 17:46 0 RDBMS DEBUG Fetched: "foobar33" 17:46 0 RDBMS DEBUG About to COMMIT transaction #<#<STANDARD-CLASS NIL {1005497791}> :begin-executed-p #t {100501B101}> 17:46 0 RDBMS DEBUG Cleaning up Oracle transaction #<#<STANDARD-CLASS NIL {1005497791}> :begin-executed-p #t {100501B101}> to database #<ORACLE {100501B001}> 17:46 0 RDBMS DEBUG Calling logoff in transaction #<#<STANDARD-CLASS NIL {1005497791}> :begin-executed-p #t {100501B101}> 17:46 0 RDBMS DEBUG Freeing environment handle of transaction #<#<STANDARD-CLASS NIL {1005497791}> :begin-executed-p #t {100501B101}> #<(AND (VECTOR T 8) (NOT SIMPLE-ARRAY)) {1003A38F1F}> * !DSPAM:4cd2e92d48581444717246!

1 0