Will Automation Hit IT Soon

It doesn’t have to be AI that does it. Forrest Brazeal writes:

No, the real trend to watch here is not that the cloud providers are making it easier for non-technical people to code (although they are), but that they are straight-up reducing the number of people required to deliver technical solutions.

I’ve been saying for awhile now that we’re getting close to a crisis point in the IT world. The mid-tier IT worker is in imminent danger of being automated out of existence, and just like with the vanished factory jobs of the last 30 years, nobody wants to admit it’s happening until it’s too late.

The IT automation apocalypse will move slowly (by tech standards), so it is flying under the radar. Unlike with the collapse of American manufacturing, we won’t get breathless feature articles and political posturing. A shuttered factory and 700 unemployed workers are concrete, easily visible; a decaying Rust Belt town makes an arresting photo spread. But how do you build a narrative around midlevel IT engineers let go in twos and threes from jobs that even they probably can’t quite describe?

Moreover, the first people to feel the pain will not be the highly-paid, conference-trotting Very Important Programmers in job-rich tech hubs. They will be anonymous Windows administrators and point-and-click DBAs and “senior application developers” who munge JSON in C#. Normal people making comfortable money, fifty to eighty thousand dollars a year in ordinary places like Omaha and Memphis and Santa Fe.

This will be the new outsourcing. Consider stuff like email, mailing lists, and simple websites. These are a hell of a lot easier to run these days than they were even 10 years ago. The marginal effort of say Google to run even a large organisation’s email is very small, perhaps not even an extra job a lot of the time.

These are good and skilled white collar jobs that will likely evaporate. And far sooner than robot lawyers take over.

Maybe babysitting predictive modes will be the new back end dev task. Maybe. Students are certainly flocking to machine learning courses.

CS departments need to think hard about this trend and what we can do for our students.

Advertisements

Ideas Not Working Out (or WILL THEY?!?!?)

Well this is a bit discouraging. I was experimenting with implementing an ELK style EL classification oriented reasoner primarily using SQLite. I love SQLite and it is super cool. I want to do more with it. If you look at the algorithms in the Incredible ELK paper you see that they involve a fair number of joins and indexes, etc. They even discuss join order and reference some attempts to implement on a database. I figured using SQLite in memory databases could be a nice win.

Nope nope nope nope. Well, not yet. The problem is that it’s way to slow to interact with the database. Consider:

43393 loops done in 1.2992407480875652 minutes
    3224564 function calls in 77.975 seconds

    Ordered by: internal time

    ncalls tottime percall cumtime percall filename:lineno(function)
    183895 70.709 0.000 75.125 0.000 {method 'execute' of 'apsw.Cursor' objects}
...
    28951 0.079 0.000 34.495 0.001 calf.py:443(in_subs)

I’m testing for whether a subsumption in my todo list is in the closure. I’m using a query:

SELECT 1 FROM subs 
WHERE subclass = ? and superclass = ?

Ok that should probably be an EXISTS, but still. It’s called a fair bit (28,951 times!).

My subsumptions here are just binary tuples of integers, so I can use a Python set to track ’em easily enough. And here’s the results:

43393 loops done in 0.6202206969261169 minutes
    2819244 function calls in 37.231 seconds

    Ordered by: internal time

    ncalls tottime percall cumtime percall filename:lineno(function)
    154944 32.796 0.000 35.274 0.000 {method 'execute' of 'apsw.Cursor' objects}
...
    28951 0.025 0.000 0.053 0.000 calf.py:443(in_subs)

Well that leaves a mark.

It’s an in memory database and I’ve dorked with the transactions and no joy.

Now it’s not hopeless maybe. Consider these:

  22592   0.320 SELECT ?, superclass FROM concIncs  WHERE concIncs.subclass = ?
  22591   0.302 SELECT type, arg1, arg2 FROM IdxConcept WHERE id=?
  22592   0.180 INSERT INTO subs VALUES (?, ?)
  14442   0.141 SELECT 1 FROM inits WHERE id = ?

Same order of magnitude of queries. Sometimes returning more data. The INSERTs into subs are pretty fast. Is it just the conjunction?

Oh crap. I didn’t have a primary key on subs. It’s compound and I forgot…D’OH! Let’s fix that and…

43393 loops done in 0.14656875133514405 minutes
         3224564 function calls in 8.811 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   183895    6.106    0.000    7.675    0.000 {method 'execute' of 'apsw.Cursor' objects}
...
   28951     0.033    0.000    0.462    0.000 calf.py:443(in_subs)

Well, that’s much better! Game on! The links query is still a bit slow:

  22592  4.977 SELECT links.subclass as E, negExists.existential as F
               FROM  (links JOIN hier as h1 ON links.role = h1.subrole),
                     (hier as h2 JOIN negExists ON h2.superrole = negExists.role)
               WHERE links.filler = ? AND negExists.filler = ?

No obvious issues. I’m unclear whether an index will help…but stilL! Back in the game!

Madoko Blues

Madoko is a very cool set of Markdown extensions/tools for writing full fledged academic “stuff” in Markdown. It compiles to HTML and LaTeX (of course!) and really has a lot of cool stuff. It supports reveal.js and beamer presentations in what seems to be a nice way.

In general, it feels like a win.

Except.

It was developed as a demonstration for the Koka language. Koka is a nice enough looking experimental language which compiles to Javascript. Ok. So any bug fixing or tweaking requires learning a new language. Ok. Well, what does it take to get started? Only this:

First install some pre-requisites:

  • Install TortoiseHg for mercurial version control.
  • Install the Haskell Platform to build the compiler (version 7.4 or later).
  • Install Node for running Koka programs (version 0.10 or later).
  • We recommend the excellent SublimeText text editor for developing Koka programs.

Then get the compiler and libraries, and build them:

  • hg clone https://hg.codeplex.com/koka (clone the Koka sources)
  • cd koka (go to the new Koka directory)
  • npm install (install needed Node libraries)
  • jake (build the compiler and run the Koka interactive environment)

Well you might as well say, “Fuck off”.

Which is a shame because Madoko doesn’t 100% work. I’ve had terrible trouble with Bibtex files and the beamer output is just awful. Often unusable. Well fine, maybe I can live without it, but it’s super frustrating when it’s right there. It doesn’t seem to generate “semantic” beamer, but maybe I can force it to? Maybe it uses templates somewhere? Ugh.

An alternative is to extract a language spec and reimplement it in Python.

(Or I guess Pandoc, but it has a lot of similar pain plus missing a lot of stuff Madoko has.)

New Slideshow System

It’s that time of year…I consider switching presentation systems. I currently use reveal.js with prez and it’s ok, but:

  1. Getting a theme that conforms to Manchester style is challenging. I’ve sorta hacked it, but it has problems.
  2. Tweaking formatting is really hard. Images in particular but really everything.
  3. Making slides progressive is harder than it needs to be and prez obliterates it.
  4. I can’t make text large enough for our screens or code listings readable

Plus I want more fancy features. I get really excited by showoff‘s interaction features (ask questions anonymously! file issues on a slide FROM THAT SLIDE!!!). But it totally defeated my attempts to put a small fucking logo in the upper left corner. It requires a ton of CSS and Javascript and understanding their framework and crises man, I just want some pretty normal layout.

I really like Makoko and it’s good for paper writing (though it doesn’t support inline footnotes because evil). It’s backed by reveal.js, but has some nice features including a very groovy editor. It doesn’t have the showoff features but it compiles to beamer. Horrible horrible and ugly beamer, but beamer nevertheless.

There are so many new HTML presentation frameworks, but they seem to be flaky and don’t solve the problem of making some simple customisations.

Updating Apps Sucks

In part because god damn UI designers like fucking with design and typically make things worse.

See the Twitter App:

I mean WHY MAKE THE NEW TWEET BUTTON A FLOATING CONTENT OBSCURING MONSTROSITY FROM HELL?! Esp since it seems to have moved from the upper right chrome to make room for the sparkle button of “home”…which takes you to some bullshit selection of tweets.

Honesty, it’s hard to imagine that they’d do worse if they were dying of some illness whose only cure spittle from my sputtering reaction.

This ain’t rocket science and it ain’t art. Just leave things alone for the most part once you have a reasonable design. Floaty things BAD!

Waterfall?

I don’t teach methodology per se in my software engineering class, at least not the explicit sequence of waterfall, iterative, and agile. I do discuss a bit about sequencing activities, the notion of a wicked problem, etc. We, in fact, iterate a piece of software.

But as with the idea of a software crises, I’m worried about our standard narratives. This blog post points to discussions of how waterfall was ever a thing and how iterative methods were proposed very early on.

I suspect that some of our hoary old narratives are just things we say. I’m not sure software is all that different from buildings or that other sorts of engineering have all that easy a time.

Fail Fast Textbook Selection

I’m strongly considering changing my reading material for my software engineering class. I like Code Complete in a lot of ways, but it really does feel a bit old and a lot of bits are not super well organised or presented. And it’s big without being super nicely modularised. It’s not really a textbook. I’m planning a pretty significant reworking of the course (to consolidate some stuff) so this is the time to change.

I was looking at a text that has some good reviews and decent presence on Open Syllabus and a metric ton of supporting material. I’ve never used supporting material but one can see the attraction!

I’m skimming a copy starting with the intro. The intro of text tends to be a really weak bit especially if it’s didactic instead of tutorialesque, so I forgave the cutesy intro dialogue. There were some helpful fake graphs about characteristic error rates which seemed fun. Then I hit the following description in a list of “kinds of software”:

Artificial intelligence software—makes use of nonnumerical algorithms to solve complex problems that are not amenable to computation or straightforward analysis. Applications within this area include robotics, expert systems, pattern recognition (image and voice), artificial neural networks, theorem proving, and game playing.

Say what? “Nonnumerical algorithms”?!?!?! Right before talking about artificial neural networks??!?! Maaaaybe there’s a specialised enough variant of “numerical algorithm” (written primarily in Fortran?!?) where this is technically not wildly false, but it sure the hell is misleading here (given the standard distinction between symbolic and non-symbolic AI). Seriously bonkers.

But…ok. Does one extremely boneheaded bit of a sort of throwaway warrant tossing the whole thing? Maybe? I want to be fair. I can always guard against this in lecture…I guess. Then I hit:

  • Software has become deeply embedded in virtually every aspect of our lives, and as a consequence, the number of people who have an interest in the features and functions provided by a specific application has grown dramatically. When a new application or embedded system is to be built, many voices must be heard. And it sometimes seems that each of them has a slightly different idea of what software features and functions should be delivered. It follows that a concerted effort should be made to understand the problem before a software solution is developed.

Oy. I mean, the dude has chapters on iterative and agile processes, so there’s some course correction. But. Come. ON!

Ok, now I dump it from the list.