Spreadsheet Errors Deep and Shallow

Spreadsheets are almost certainly the most common form of end user programming. And they are both cool (enabling amazing systems to be built by all sorts of people) and dreadful (those systems are even more buggy than we fear). There’s lots of cool research on them…even inside Microsoft!

But I’ll tell you right now that the thing that fucking kills me with Excel for Mac is that scrolling is just the worse. It’s slow, buggy, and just plain hard to use. I switched to Numbers for dealing with a spreadsheet of less than 10 columns and less than 200 rows just because the scrolling was KILLING ME LOUDLY…

…of course, Numbers doesn’t let you delete specific cells…only whole rows why for the love of every sort of sanity on this good green earth WHY?!?!?

It just boggles me that these simple things are so broken. For generations of versions.

April Fools!

A PhD student that I let hang out in the group office next to mine likes to play pranks hence the ballon related one that greeted me this morning:

How they have the time to work on their PhD escapes me.

I, of course, launched a counterprank by emailing them that it was a fire safety issue and that there were escalating consequences. That was working until one of my PhD students pointed out that I had done that trick on them. The original prankster was only half fooled. Boo!

So I ran the same prank on the snitch only adding a forged email from Health and Safety for added verisimilitude. Worked a treat!

We are Very Serious People at the University of Manchester.

Apple Store Suckery

Zoe’s phone died. It was doing the wonky battery thing. This was one of the recalled iPhone 6Ss. Last year I had the battery replaced under the program. It didn’t quite fix the problem but did seriously mitigate it.

Ok but now it won’t charge. As the Apple Store guy pointed out…it rattled.

Yikes! I guess they did a crap job last year.

Nope! It was a knock off battery poorly installed.

Now of course they didn’t “accuse us” of anything…they just thought that they fixed it properly and sometime in the past year, probably the past month, someone meddled with it and so no fix for us.

This phone has literally never been out of our possession. We had no motive to have any other repair or battery repair…we thought it was still under warranty!

And don’t fucking tell me that you aren’t accusing me because you did it by implication. That just sucks. Just say we can’t comment on the mechanism but since we can’t prove it’s their fault we’re shit out of luck.

Of course this doesn’t just mean they won’t repair it for free, it means they won’t repair it.

There was an iSmash in the mall that repaired it in under 15 minutes while we watched. Apple drove us to what they forbade.

We’ve dumped serious cash into that store. I don’t have a good feeling about them.

A Sense of Proportion

Just wanted to pull up this comment about Louis CK’s return from non-exile. The impetus is this tweet:

My snark:

I too am concerned about released felons who find reintegrating into society enormously challenging (from mental health issues, to getting a job and voting, etc.) Registered sex offenders, while often—but not always—monstrous often find it literally impossible to live in a given area because the imposed restrictions they face make it so there’s literally no place where they can legally stand. This is super tricky when we’re talking about the monstrous ones, since they did do, after all, horrible things. On the other hand, when we release them there should be some way to function in the world. This is before we get to the incident of unjust convictions, etc.

Figuring out a path that is safe for society and allows reintegration is challenging. Redemption is even trickier though there’s amazing work by the restorative justice folks that’s worth exploring.

Wait, Black is talking about how to let rich white guys get the lucrative spotlight again?

Never mind.

Seriously folks, esp people with a significant platform, before defending some rich asshole, please spend a few minutes thinking about whether there’s some actual problem you could talk about.

With respect to the #metooed, how about thinking about the lives they’ve blighted? That’s a good first place to direct your attention.

Update: Hana Michaels wrote an excellent article about this seamlessly blending her personal experience with broader analysis. Read it.

Some Annoyances

Welp, I lost my UK passport on my last trip. I finally faced that fact and cancelled it. Now I’m going through the pain of renewing it. This means I won’t be able to the US next week because everything sucks. Apparently you can’t do online or super expedited passport renewal for lost ones, only “one week” fast track. There’s no reason at all given. It makes zero sense. In spite of them making much of giving me a reference number for my cancelation, none of the 3 or so distinct application processes ask for it.

And the web site. Ugh. I’ll just detail the final annoyance which was  the payments form:
Without spaces?!?!? Duuuuudes! It probably took more effort to write “(without spaces) “than to strip the fucking spaces.

(If you ignore that as one is prone to do then it tells you that there was an “error in processing your payment” thank you very much. At least it doesn’t dump everything you’ve filled in to that point like other bits of the site.)

Making Barcharts in Python is Way Too Hard

Use mathplotlib! Ignoring all the articles talking about what a pita it is!

It didn’t install. I think it might already be installed. But this sucks.

Ignore all the libraries that generate JavaScript thingies. Or require GNUPlot or similar.

That leaves Pygal or Leather. They both generate SVG. Which LaTeX and many Markdown2PDF tools don’t like. But Pygal will generate PNG!!! Yay!

Which needs you to install cairosvg. Ok.

I get this to work! Yay! No way my users want to do this so it’s off to pyinstaller. Which horks on Pygal AND needs special handling in virtual environments which I am using because it’s supposed to be virtuous.

But now I have a single file executable!! I’m the man!

Which doesn’t work on Sean’s or Uli’s machine for random reasons. I figure out that pillow 5.0.1 won’t work on Sean’s machine because it’s MacOS 10.11 which the pillow folks borked because Apple. So I downgrade pillow which makes my program not crash on Sean’s machines but now the graphs are randomly borked.

Sean tries to run the script directly but his Python is funny. It doesn’t work for Uli at all because who knows? (She has MacOS 10.13.) Oh, and sometimes dragging something from one Dropbox folder to another on her machine is interpreted as “replace that file with a same named file filled with random HTML”.

So I give up, get the data, and generate the report on my own damn machine. Which reveals some bugs. Which I fix.

I then decided fuck this shit and went to a movie.

Python might have batteries included but they will shock you and leak acid into your tea.

Universities Are Not Well Run

I read the news today, oh boy:

Students planning legal action against University of Manchester for lost teaching time during strikes

It’s understood that during talks between the student union and Manchester University, one suggestion has been that graduation fees – of £35 – could be waived in compensation for lost teaching time.

Remember that the people who were on strike are getting their pay docked. That is, the university feels that if we do not supply what they are paying us for then we don’t get paid on a pro rata and proportionate basis.y rough and low bally estimate is that students pay £50/day.

Now, to be fair, it’s complex. We didn’t shut down the whole university. But even then, it’s a bad look.

And nominal fee isn’t the whole story…it’s what’s been agreed upon for that fee.

Given that this whole thing was utterly unnecessary because the precipitating event was absurd, evil, and based on lies by the universities, they should step up.

But perhaps they are hard up?

Latest figures show increases in universities’ income, surpluses and reserves

UK universities’ income increased by £915m (2.7%) between 2015/16 and 2016/17, according to figures from the Higher Education Statistics Agency (HESA). During that time they made a surplus of £2.3bn and now have total reserves of £44.27bn.

The data also revealed an increase in capital expenditure, but that the proportion of money spent on staff had not improved. Compared to seven years ago, the percentage of expenditure spent on staff has fallen by 3.35%, but the percentage spent on capital expenditure has shot up by 34.9% over that period.

UCU said the figures made a mockery of universities’ claims that staff were a top priority. Staff in universities have seen their pay fall by around 20% since 2009,

So no. They just don’t care about spending money on staff.

I know that in some circles having shrinking staff salaries in a labor intensive industry would be thought to be good management. But those circles suck. They don’t care about the labour force.

And then there’s this lovely bit:

On Wednesday 25 April Senate endorsed the Faculty Leadership Team’s recommendation regarding a change to the School structure in the Faculty of Science and Engineering. Senate endorsed the proposal to reduce the number of Schools, from nine to two and name the new Schools the School of Engineering and the School of Natural Sciences.  In addition, the paper submitted to Senate included the following recommendations which did not require formal endorsement:

• the retention of the current School disciplines in ‘Departments’

• the formation of a Faculty Teaching College and a Faculty Research College.

Yes, a reshuffle that essentially is one part renaming and one part adding an extra level of management is a bold response to the many challenges we face! This makes me really confident that our senior leadership team is trying to acquire managerial cred for their next job…er…solve the tough problems…er…help US solve the tough problems of teaching and research in these troubled times.

A Bad Paper about Federalism

Whoof. “Federalism and the End of Obamacare” has been sitting in a tab since this summer. Time to read it!

Let’s start with the abstract:

Federalism has become a watchword in the acrimonious debate over a possible replacement for the Affordable Care Act (ACA). Missing from that debate, however, is a theoretically grounded and empirically informed understanding of how best to allocate power between the federal government and the states.

Well…maybe? I’d like to see some evidence that that understanding is necessary…or missing…or desirable. Do we need to know how to “best” allocate power between federal government and the states? Best in what sense?

For health reform, the conventional arguments in favor of a national solution have little resonance: federal intervention will not avoid a race to the bottom, prevent externalities, or protect minority groups from state discrimination.

Well…preventing insurance selling across state lines helps with a race to the bottom. And health care was already significantly nationalised cf Medicare, Medicaid, and, you know, the employer health care deduction.

Instead, federal action is necessary to overcome the states’ fiscal limitations: their inability to deficit-spend and the constraints that federal law places on their taxing authority.

I think this is sorta reasonable. Enough on the abstract…it makes me a bit wary, but not despondent.

At the core of our federal system is the principle that the states should take the lead unless there is a need for national action.

No. This is tendentious. I’m not even sure what “take the lead” should mean. The core of federalism, if anything, is  that there is a division of power between the federal government and the states and this division is inherent. That is, it’s not a modifiable “gift” from one to the other. The UK government is not a federal system: the UK parliament is supreme and devolution of power is voluntary and revokable. Parliament could claw back all the power devolved to Scotland by mere majority vote.

Now, obviously, it’s one vision of whywe have federalism is that lots of things are better handled at the state level yadda yadda. The federal government is free to, with its own power, to act more nationally (a la Medicare) or devolve some to the states (a la Medicaid). The latter is definitely more in the spirit of “laboratories of democracy” vision of how US federalism should be. But that’s not a direct aspect of US federalism.

Federalism is said to foster political participation, to enable experimentation, and, especially, to allow states to tailor their laws to better suit the preferences of their citizens.

And screw other states or exclude large swaths of its citizens from political participation. Plus, when we look at states like California…c’mon. It’s huge. If it is “tailoring” so to can the federal government.

Purely as a strategic matter, the emphasis on federal law needs some defense. By way of comparison, consider same-sex marriage. When Massachusetts eliminated its prohibition on same-sex marriage in 2003, advocates did not turn immediately to the Supreme Court. They built the groundwork for a national strategy by winning in state courts and state ballot boxes.

Oh come on! You have to work a case up to the Supreme Court. And Lawrence v. Texas was in 2003. But also, you had Bush in the White House thus strong White House opposition. As this time line shows, there was a lot of legal activity, some of which made it to the Supreme Court. The fight over DOMA was federal!

My point is that this throwaway line presents the fact that there was a 12 year gap between the 2003 MA decision and the 2015 Obergefell decision as if it was a deliberate choice by activists with a suggestion that they were motivated by “federalism” concerns.

Aside from this almost certainly being false (there was no monolith, people were primarily motivated by tactical and strategic considerations, to a first approximation, all advocates of gay marriage would have been fine with a federal solution as early as possible.

And really, is it federalism if the states startsomething but then the feds (e.g., the Supreme Court) imposes it uniformly? What variance is possible then? How are we tailoring things to the local populations.

In any case, marriage is weird because marriage was traditionally something managed by the states. You get your marriage license from a state, hence going to different states to avoid annoying regulations in your home state. Federal involvement primarily is: All states must recognize all marriages from other states and the federal government makes uniform policies on all the diversely regulated marriages.

My point is that this is a shitty example, with shitty backup, presented in a shitty way. This is NOT what I want from someone purporting to give a careful theoretical and empirical analysis of federalism.

Given that I’m not professionally obliged in anyway to read this, I’m going to stop. Maybe there’s gold later on, but with this much upfront  garbage, I’m out.

Software Gripes: Scrivener and ConcertWindow (and WordPress)

I think I need regular “features” i.e., columns of a particular type or theme, to keep the blogging going, so here’s a new one near and dear to my hard: ranting about software problems (I’ll through in other system gripes but the most common is software).


I want to love Scrivener. It certainly is enticing, if a bit complex. I’m trying to use it as a course materials (lectures, quizzes, etc.) management and editing tool. People certainly seem to have had some success with it as such. I think it could also be handy for paper or book writing and esp. grant writing. Grants have VERY complex and finicky structure which Scrivener’s “break it into bits” and “annotate and organise” and “hey, templates all the way down” approach looks to be quite good.

But there’s a fundamental problem: The whole Scrivener model is “compiling” the project into a single final document. Really. Uhm…that’s bonkers. Even if your final output is conceptually a single book, you very well may want the “out of Scrivener” view to be split up in multiple files. (Think Website with a separate HTML page per Chapter. Or just Website.)  For courses, I don’t want one output to contain it all, I want lots of documents (syllabus, references, slides broken out by day or by lecture, quizzes, lab sheets, etc.) Scrivener HAS THAT STRUCTURE, but, as far as I can tell, it doesn’t like to spit it out. You can “export” the file structure, and maybe that will turn out to be good enough. (I only figured that out today.) But I want some of the structure to be flattened! E.g., if I make a Lecture which has separate subdocuments as “slides”, for some workflows they should be combined! But the whole Lecture shouldn’t be combined with all the other lectures. (Except for the global print version.)

Ok, “export” at least lets me write my own custom compiler. But then why do I have to deal with the “project” structure and explicitly set “export”? Why can’t a project just be a directory/file structure in the file system. In other words, why “export”? That adds a really painful step to the process. It makes synching harder, etc.

Additionally, Scrivener has some simple WYSIWIG formatting (bold, italic, tables, lists, etc.) It also has export to MultiMarkdown. This all seems extremely promising for downstream processing: Write using the GUI, explore to Markdown, then run tools that parse and manipulate the Markdown to generate the final versions.

Oh, silly me! All Scrivener does is compile snippets written in MultiMarkdown to other formats (HTML or PDF via LaTeX)! You have to write the Markdown.

Well that sucks. It’s not like Scrivener is a word class Markdown editor with syntax checking etc. The key formatting features it supports in the GUI are eminently Markdownable, so why not export to it? Indeed, for things like Tables, having a reasonable GUI is much much nicer than hacking Markdown syntax directly. Sigh.

Finally, they have this cork board view. Before 2.7, it defaulted to a cork textured background and index card looking cards. Very skeuomorphic, but in a good way. It took you out of the UI and forced a cognitive mode shift. 2.7 it defaulted to a “flat” interface that was 1) bland and 2) merged it visually with every other view.

Sigh. But wait! You can tweak it back. But now, in my preferred Index Card style, they stuck a pushpin.

screenshot_03Why, why, why, why?! It doesn’t read; it doesn’t help; it forces a “vertical” orientation (I actually viewed them as piles before). This little tumour does exactly nothing positive. It serves no visual-informatics purpose and, indeed, distracts. It’s centred, bright, and in line with meaningful information. This is skeuomorphic madness, where the designer slavishly emulates the real world object without thinking about the design. Pushpins are not a useful information part of the design…they are there to hold the cards in place. If you lay the cork board flat, you don’t need them.

“But Bijan,” you say, “the cork only exists to have pins pushed in! Isn’t that the same problem?”

No, gentle reader, while the cork in the real object is there functionally to be stuck with pins, it has several user interface functions: 1) visual mode switching; it’s a very strong cue about the difference in working style; it provides an information cue, 2) it supports the illusion without affecting other information per se, and 3) it is high contrast yet not obtrusive. The main problem with skuomorphism is that people take it too far. The idea shouldn’t be to exactly replicate the real world object, but to design an interface that works. Flat interfaces general suck because they generally designed that chrome should be indistinguishable from content (or not be perceptible at all) and content should have few sub distinguishing features. (Microsoft’s Metro interface is something of an exception.)


Zoe tried to do a ConcertWindow concert last Sunday. There were numerous technical hassles, but we managed to struggle through most of them and have a reasonable concert which most viewers could see most of. One cool feature is that you can get the full recording of the stream and the website lets you post a one song snippet of the recording on their website. This was exactly what we wanted to promote the new album (in progress).

We do not have such a recording.

The reason we do not have one is that they have a “feature” that is supposed to help you debug your streaming. For a given concert slot, you can set up a “test” session which will not be exposed to anyone except your testers and can happen at other times than your scheduled slot. This sounded sensible, but there were a few problems:

  1. It doesn’t work from the iOS app, which is how were were going to broadcast the concert. Grr. But ok, we can at least test the basic setup via the browser version.
  2. Testing via the browser version just doesn’t help very much. You still need to test via the iOS app. A lot. So we were scheduling test concerts all over the place. That was better in someways, since that’s what exposed that the “Pay what you want” option is really “Pay what you want as long as it is at least $1”. Grr.
  3. When you go to look at your video, the prepend “for your reference” all the test video you did. What? Why? Who wants that? Who wants that in their concert recording? Shouldn’t you just save that as a separate file, if at all? Weird.
  4. Oh, and if you tested in your browser, but recorded from iOS, you now have a video that is half test video and half corrupted nothing. That’s right, the “test” mode can corrupt your concert recording. So we have no video of the concert, whatsoever.
  5. In the FAQ for “Preparing for the show” they have “How can I sound check before the show?” which says

    Choose if you’re going to broadcast with Web, iOS, or RTMP, then switch to “Test” mode and start broadcasting. No one will be able to see it on your channel. Click the “Test URL” link below the broadcaster and you’ll be able to see your test stream in real time. You can also send this link to a friend.

    In the FAQ for  “After the show” they have “My archived video file has errors and/or the recording is corrupted” (it’s on the SECOND PAGE of this FAQ)

    This can sometimes happen if you broadcasted to the same show via multiple devices (iOS + laptop) or in different frame rates / formats.

    To avoid this happening, be sure to broadcast to each show using only one device and one video/audio format.

    If you do broadcast using multiple devices or formats, the live stream will work totally fine, but the archived recording may be corrupted.

    So, the advice they give before hand can corrupt your recording because they have a feature (prepending test video) which is completely worthless. And their own help leads you there.

Message to the ConcertWindow programmers who did this: Never corrupt important data. Never. Ever. Especially don’t corrupt real data with test data. I mean…come on. Shame

Message to the ConcerWindow documentation writers who did this: If there is a risk of data corruption…DON’T RECOMMEND ACTIONS THAT RAISE THAT RISK. Oh, and WARN PEOPLE ABOUT THE RISK AND HOW TO MITIGATE IT before they might do the action that destroys their data.

You should be profoundly ashamed of yourselves.

While we’re talking documentation nonsense, let’s consider this gem:

At Concert Window, we give the artist a full private copy of their show, for free. You can use it for any non-commercial use, including uploading it to YouTube. 

The video files are in .mp4 format, which is playable with most major video players including VLC and can be imported into iMovie and Final Cut Pro.

Sometimes, due to errors during broadcast or other reasons, the video files may be corrupted or unplayable. In that case we’re sorry but there’s nothing we can do. This is part of why we offer video archives as a free service.

In addition to downloading your full show recording, you can also create a short highlight video. Here’s an article with more details: How to create a highlight video

*Artists are not allowed to sell their show videos due to copyright restrictions.

First, note the “or other reasons” for corruption…like BEING MISLED BY THE DOCUMENTATION TO HIT A DESIGN BUG WHICH IS KNOWN TO CORRUPT YOUR CONCERT. Maybe you should fix that.

Second, note the nonsense of the highlight blocks. Zoe owns the copyright for the songs she played and the performance. The terms of service explicitly SAYS that she owns the copyright.

(BTW, the terms of service are absurd and horrible. I’ll break that out in another post.)


Current gripe: Adding a category doesn’t put the new category under the parent one you’ve selected.

Also, I want to have categories be more meaningful. I’m currently inserting two key categories into my post title (see current post’s title): Music Monday and Software Gripe. This is wrong. I’m polluting my title with Metadata about my post in order to get the visual effect I want. Boo!

MLK day-after irony

Of course, the UK doesn’t celebrate MLK day (which was yesterday) so I was at work. Boo! C’mon UK! As the 51st state, you should get with the program!! Or programme, even!

My MLK library is scattered across two continents with a big chunk in Swarthmore, so I decided to get an ebook copy of my favourite of his books, Where Do We Go From Here: Chaos or Community? It’s a great great book and a useful antidote to Teddy Bear Having Dream MLK that some people will throw at you (universal basic income? it’s in there!). The title alone is worth contemplated: It does feel often that the (good) answer to the threat of chaos is community. But community is extremely difficult to achieve in most societies esp. with big legacy hatreds. The question of avoiding chaos is a fundamental question of political science and a lot of the answers aren’t good.

In any case, Amazon (boo!) does have it as a Kindle book. It’s not super cheap, and I wish someone would pay the MLK estate to make all his books public domain. Or they would just free some up. I want more people to read these.

The irony came from the Amazon algorithms:

On the page for MLK's book, there's a recommendation for Trent Lott's book "Crises Point".

On the page for MLK’s book, there’s a recommendation for Trent Lott’s book “Crises Point”.

Trent Lott? You’re going to recommend a book by Trent Lott? Ok, coauthored by Tom Daschle (and a ghost writer), but still. None of these schmucks are worth recommending by themselves (much less Yet Another WankerFest About Broken Politics By Third Wayers), much less for their instantly forgotten book.


Read MLK! Read Letter from a Birmingham Jail if nothing else.

Read the good stuff. Life is too short for the crap.

(I also have Dyson’s April 4, 1968 up on Scribd. I find Dyson worth while but a decidedly mixed bag. It’ll be interesting.)