Making Principled Unprincipled Choices

I like principled decision making. Indeed, few things inspired me as much as this quote from Leibniz:

if controversies were to arise, there would be be no more need of disputation between two philosophers than between two calculators. For it would suffice for them to take their pencils in their hands and to sit down at the abacus, and say to each other (and if they so wish also to a friend called to help): Let us calculate.

Alas, there’s no decision making situation where this vision holds, even in principle. But still, I like my decisions to conform to some articulable rationale, preferably in the form of some set of general rules.

But some of my rules are meta-rules which focus on resource use. Obviously, one goal of decision making rules in to maximise the chances of making the “right” choice. But for any metric of rightness (let’s say, an appliance with the best value for money) there’s a cost in the effort to assure the maximum (e.g., research, testing, comparing…lots of shopping). That cost can be quite large and interact with subsequent satisfaction in a variety of ways. I’m prone to this and, indeed, end up in decision paralysis.

In response to this, one of my meta-rules is “don’t over-sweat it”. So, for small stuff, this reduces to “don’t sweat the small stuff”. But, because of my anxiety structures, I tend to see certain classes of small stuff as big stuff. So, I dedicate some effort to seeing small stuff as small. Sometimes, this means making it invisible to me. Poor Zoe often has to make the actual purchase after I’ve done the research, or even make the decision after I’ve done the research. For various classes of minor, irrevocable sub-optimal decisions, I prefer not to know about them. I will obsess, and that doesn’t help anyone.

When the decision is essentially arbitrary (because all choices are incommensurable in toto, or their value is unknowable at the moment), I try to make myself flip a coin (metaphorically, at least). What I try to avoid is building a fake rationale (except when that enables the choosing or makes me happier with the arbitrary choice).

Technical (or teaching) decisions often are best treated as arbitrary, but we have tons of incentives to treat them as requiring a ton of analysis to make the “right” choice. At the moment, I’m evaluating what Python testing framework to use and teach in my software engineering class. I currently use doctest and unittest and have a pretty decent lesson plan around them. doctest is funky and unittest is bog standard. I’d consider dropping doctest because I need room and we don’t do enough xUnit style testing for them to really grasp it. They are also built into the standard library.

But then there’s pytest which seem fairly popular. It has some technical advantages, including a slew of plugins (including for regression testing and BDD style testing). It scales in complexity nicely…you can just write a test function and you’re done.

But, of course, it’s a third party thing and needs to be installed. Any plugins would have to be installed. Is it “better enough” to ignore the built in libraries? Or should I add it on with the builtin libraries? AND THERE MIGHT BE SOMETHING YET BETTER OUT THERE OH NOES!!!!

No. The key principle here is a meta-principle: Don’t invest too much more effort. Make a decision and stick with it. In the end, any of the choices will do and a big determiner will be “does it spark my interest now?” while the other will be “how much extra work is that?”

And that’s fine.

 

Advertisements

The Cyber Security Body Of Knowledge

This effort looks cool. Providing a codified overview of What We Should Know about cyber security could be very helpful, esp for teaching. I just read the Software Security Knowledge Area and it wasn’t a bad read. It felt a little “listy” without a good cognitive map. In particular, the cop out on the completeness of their taxonomy of faults. I don’t blame them on not claiming to be comprehensive, but I don’t know whether they cover the bulk or the important ones just from reading this. I should have a better sense of what I don’t know!

Then there was this thing that bugged me:

• A detection technique is sound for a given category of vulnerabilities if it can correctly conclude that a given program has no vulnerabilities of that category. An unsound detection technique on the other hand may have false negatives, i.e., actual vulnerabilities that the detection technique fails to find.

• A detection technique is complete for a given category of vulnerabilities, if any vulnerability it finds is an actual vulnerability. An incomplete detection technique on the other hand may have false positives, i.e. it may detect issues that do not turn out to be actual vulnerabilities.

Oy! This reverses the ordinary (i.e., mathematical logic) notions of soundness and completeness…sorta. They didn’t quite flip the meaning, but instead they focused on an entailment class that’s weird. Take soundness. The entailment they pick is, “Program P, for vulnerability class V, has no Vs.” It’s that “no” that messes it up. In order to conclude that there are no Vs it has to be the  case that ifthere was a V, it would find it. I.e., that it was complete with respect to “P has a V”. And I mean, the last sentence makes it clear that they are thinking at the “P has V” level. And, of course, their bogus complements focuses on the “P has a V” level, so they just screwed up. Sigh.

It would be much more straight forward to define a detection technique for V as a procedure which takes P as an input and returns a list of “P has a V” statements (with specific Vs). Then the technique is sound if it produces no false positives and complete if no false negatives. A sound and complete technique that returns the empty list allows us to conclude that the software is secure wrt V.

Then there’s this:

It is important to note, however, that some detection techniques are heuristic in nature, and hence the notions of soundness and completeness are not precisely defined for them. For instance, heuristic techniques that detect violations of secure coding practices as described in 2.3 are checking compliance with informally defined rules and recommendations, and it is not always possible to unambiguously define what false positives or false negatives are. Moreover, these approaches might highlight ’vulnerabilities’ that are maybe not exploitable at this point in time, but should be fixed nonetheless because they are ’near misses’, i.e., might become easily exploitable by future maintenance mistakes.

Sigh. Detection techniques that are “heuristic” are generally unsound or incomplete. What they seem to be talking about is problems (or maybe just infelicities) with the definition of some category of vulnerabilities.

Still! It’s in development and even as such, I’d point a student at it. These things aren’t supposed to substitute for textbooks, but they can be helpful as a quick orientation and sanity check.

C++ Header Analysis

I’ve been running a third year project on “analysing the Python ecosystem.” It was intended to be pretty flexible (eg analysing available teaching books for coverage). I’ve had various nibbles and bites at both the 3rd year and MSc level, but people find it confusing. I keep thinking it should be easy given how much stuff that’s easily available but people don’t seem to know what to ask.

Here’s a pretty simple analysis of C++ headers. As an example, it’s ok. But it does suffer from the standard problem that it stops before any interesting questions get answers. What I do find interesting is the way that C++ ends up importing vector all over (since it’s super useful but not built in). But it’s not a very surprising or deep result.

Electronic Kanban Boards

I’m a fan of Trello.

Up to a point. I’ve made many a board and found it super useful until my efforts to use it inevitably peter out. But that’s probably me.

Kanban boards seem worth teaching in a software engineering course if only because they are so common. They are also fairly easy to grasp so you can have a pretty good discussion about organisation and management. There’s a lot of intellectual baggage associated with them but just getting students to think about writing down todo lists is a win.

I doubt I can force a class to sign up for Trello. Requiring students to use third party services has a lot of implications that were dodgy before the new data protection laws.

This article discusses five open source alternatives but frankly, even though there are some attractive feature lists, they don’t really appeal. Worst, the idea that I would either host such a thing or get some other part of the uni to host it seems impossible. I guess the students could run the one with docker images available…but that seems more impossible.

We can go back to post it’s or index cards…but we don’t have a shared environment.

So…maybe it’s better to focus on issue lists and bug reports? Keep off these larger management issues until the next class?

When the Web Does Your Coursework

In my software engineering class, at one point they have to refactor their command line app (in Python) from handling all argument handling by hand given nothing but a list derived from the invocation string to using the built in library argparse. They then have to write an essay comparing the two and providing a technical recommendation. It’s a great set of interlocking coursework (and lectures).

Kyle Purdon has a really nice blog post comparing Python command line libraries. (There are others!) Of course, it’s not exactly my essay assignment because I saw Kyle’s post (and others like it) and was worried about confusion and academic malpractice. But perhaps I overthought it? Maybe I could find a library not covered and ask them to extend the analysis?

Most of my students, I’m guessing, don’t look for stuff like this anyway.

Perhaps I should keep my essay the way it is, point them to Kyle’s post, and then have them revise it?

This is a very complex era to design coursework in.

More Spilled Ink

I’ve made some progress playing with ink since my first post on it. The inky editor is pretty ok for small things (thus far). Freb wrote a story in the soon-to-be-defunct inklewriter which they graciously let me port over to ink and mangle in a bunch of ways. (Inklewriter does not export to ink, thank you very much.) (Oh, the compile cycle with inky is…not terrific.)

ANYWAY, I managed to try a few different features including variables, conditional text, random text, etc. (see this version). I planned to port that to a few other IF systems for a head to head comparison but I decided that it would be better to write a series of versions gradually introducing new features. That way, I’d have fodder for a tutorial and I could port simpler things as I learned the new system. Check out the Github repo. I need to get CI with publishing to Github pages working. There are a ton of finicky bits!

“Actively Maintained” Claims Need Active Maintainence

Software is hard. Free (both as in speech and as in beer) software is hard and often not well compensated. I’m going to whine a bit about very free libraries which tends I bad form. I first want to sincerely thank the people who work on these libraries! You have my admiration and appreciation even as I criticize.

My learning experience on iOS (and specifically iPhone) based python development is extracting the results of the Twitter searches around “The President Sang Amazing Grace” and, I was hoping, to do some simple analytics. Maybe eventually do some sentiment analysis. In principle, you’d think the interesting bit, even moderately challenging part, is in the analyses. Per usual, just getting things up and going is a big PITA.

Out of the four libraries I found with a quick search, I settled on Twython, in spite of the name, because it had the nicest (still not complete) discussion of authentication. In particular, they clearly demonstrate the principle of least privilege by emphasizing the auth level sufficient for read only access. Since this is a large class of Twitter apps, I think it’s important to be emphasized. Yay! It also says that it’s actively maintained! Yay! Unlike those other loser libraries, I guess!

But searching Twitter is a pain. If you have a large number of Tweets in the results, you have to paginate. Furthermore, there’s a question as to how far back you can go (some search specific libs say 7-10 days is it, but the Twitter api seems to have some date range functionality).

In any case, Twython, like other high level libraries, provides a cursor abstraction. But the docs don’t say exactly what it abstracts over! Will it get the next page transparently? Do you need to be in page mode?

But worse, we have a bug of many years standing (with a stale pull request!) which causes a cursor to freaking loop on search results under some conditions. Well that seems to make cursors utterly useless (maybe they are useful out of box in some cases I don’t know, or maybe everyone writes loop detection code in their cursor loops).

Regardless, it makes the “actively maintained” claim…less than true. At the very least a note in the documentation is required but really this bug should be addressed if “actively maintained” is to have any meaning.

(It goes to show that one really REALLY has to work through any coursework one hopes to give. Reading the documentation is not sufficient!)