MyFin.doc A raison d’etre

Beginning  around 2003 I started a Journey in Cyberspace that has no reasonable conclusion, and resulted in no real (patentable) Intellectual Property being created.

At that time in Highland Heights KY i found myself jobless after many years as a computer/Engineering professional better described here

I knew very little about the mechanics of how the World Wide Web actually worked. I did however know a little Perl, and had access to one of NKU’s minor servers wholly own by a rather radical bunch of folks, and befriended this man

Their server (solaris 5.1?) zappa.nku.edu and my home machine a windows 95 laptop perloined by my Ex from  AT+T Lucent Avaya (who can remember?)  became a first home of A set of comma delimited files (similar to these today).

Perhaps it was this book Clinton Wong that allowed me to create those early datasets from data that Yahoo (at that time) so freely distributed. The first activity involved loading this data into a Microsoft Access DB, and writing snatches of perl (mdb2html.pl) to create html (version 3 or 4 i’m sure), and then Post these static web

pages on zappa. In the mean time i am becoming familiar  with ActiveState Perl, and the many modules i can use to leverage tasks like table generation, file parsing etc.   Heavily relying on John McNamara’s work I did have the beginnings of what would become dynamic html created statically on that Win95 home machine as Excel Files like this

The front end to this system became Perl Processing this raw data statistically, and then rendering through colorization the data for display. Even given the creation of these new Statistic File Summaries a significant amount of Perl data manipulation uses that data as input to a “Ratings Engine” rewriting that Ranking Data back out to these csv files (merge.pl EXTRACT & tabstats.pl TRANSFORM ysplit.pl LOAD)

It was only much later that it became apparent that CSV, and Text files i was creating using LWP and ETL perl procedures were in fact  2  Flat File DBs. MySQL the emerging relational db (yes i had done some oracle) technology were NOT available (at least working versions) on zappa so it was around this 2004 time frame i became the defacto Ux Admin on Zappa. In fact some professor unknown to me started using my instantation of that MySQL db to teach CGI programming which lead US to the parting of ways (2008?). Not to mention that even at Solaris 5.3 Sun didn’t have good support for things like php5 apache II or MySQL which is ironic since they `​owned` it until Oracle acquired the remnants of Sun which i suppose includes Technologies like Tcl/Tk.

I do remember giving one of those Math Talks where i showed (off) some of this software, and how holes in zappa’s CGI infrastructure `​might` allow one to surreptitiously examine root’s passwd file. Then a few years later some Russian (apprarently) hackers’ implanted some Gambling Website advertisements onto this Sun server.

MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. Being a fork of a leading open source software system, it is notable for being led by the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle.

So with the Help of  phpMyAdmin

The next generation of this software used a real relational db along with dynamic html as displayed here . Note that Oracle’s 185B Market Cap Mammoth had to swallow the pitiful MySQL (Ag). leading the way to something called MariaDB, but again i digress. If you examine the link above you will see the underlying method is csv2hcol.pl and friends. The Excel methods are and still remain pretty much as originally developed under win95 which speaks volumes to the portability of perl.

The changes usually involved broken links by the ways Yahoo had of changing how they displayed their data.

It was around this time (2007?) that Solaris, Zappa, and I had to part ways. The next port was to something like Fedora Core 12 (Linux). Unfortunately my rent on this new space was even “more expensive” in that i was constantly getting time sucked away into maintaining these same machines as  mail servers yeah Exim, Sendmail, and then Postfix. Of course that was good learning time for more perl more php more mysql…

So  the Files were actually the DB ?. In the Perl sense one can treat flat files with SQL select statements (ReadOnly),  but i suppose Yahoo was never really my friend in all this . At some point they never allowed CSV files for download anymore. Me ? no problem i wrote an Html parser to scrape KeyStats HTML pages. The root issue of course was the numbering scheme of the csv files themselves. Roughly speaking Yahoo had decided on 10 Sectors, which contained something like 216 csv Files OR if you like Industry codes. Oh it would have been so Nice if they had only used SIC or NAIC codes !!

Fast Forward to 2016 Yahoo totally changed their page styles, and i have been stymied trying to parse these new pages even try ing Python’s Beautiful Soup for example. So I spent roughly 2-3 months reinventing the data acquisition scheme to use JSON. Then YooHooFinance decides to pull their Industry Browser which is  how i was getting a catalogue of stock symbols to pull data on.

This revealed my total dependence On the YahooFinance data schema, but rather than giving in I now go to Nasdaq for this symbol data which is now the FrontEnd (FrontEnd) to a process (industry.php) that now involves Fitting the Square Peg (S+P 12 sectors, 216 Industries) into the round (w)hole (Yahoo 10 Sectors & 200 industry codes) which were always opaque to the outside world anywayZ.

The Saga will continue, but the Original Software Development Objectives are already somewhat documented

 

 

 

Advertisements

%d bloggers like this: