Debugging iOS Safari

Ever have a rendering issue on iOS browsers while you’re working on a website/web app?

Here’s a neat trick that I’ve taught a few people over the past couple weeks: you can use desktop Safari to help debug issues that arise on iOS Safari. You can do this on the iOS Simulator, too, which can help you debug rendering issues on iPhone and iPad, retina and non-retina, even if you don’t have one of those devices.

iOS Simulator

To do this using the iOS Simulator, you’ll want to download and install a recent version of Xcode, first.

To get to the iOS Simulator once you have Xcode installed, open up Xcode and then in the top-right "Xcode" menu (next to the "Apple Menu"), go to Open Developer Tool->iOS Simulator. After a few seconds of loading, an iOS home screen should appear.

Hint 1: You can change devices and iOS versions by going to the Hardware->Device menu. (If you want to test older versions of iOS, go to Xcode, then open the preferences — Xcode menu->Preferences. Then under the "Downloads" tab, you can choose to download various iOS simulators under the "Components section".)

Hint 2: you can trigger the "home button" action by pressing ⌘⇧H, command-shift-H.

In your iOS Simulator, open Settings and go to the settings pane for Safari. Under "Advanced", make sure "Web Inspector" is turned on.

Now open Safari in your simulator and browse to whatever page you wish. (Since the Simulator runs within your own computer, you can even debug sites under without having to worry about port/IP address permissions — it’s just another local web browser.)

Open your Mac’s desktop Safari. In preferences, go to the "Advanced" tab and make sure "Show Develop menu in menu bar" is checked.

Now, under the "Develop" menu, you should see "iPhone Simulator" or "iPad Simulator" a few options down from the top. Select the tab that corresponds with the page you have open and you'll see the standard Safari web inspector acting on the browser in your simulator.

iOS Device

Now, for an even cooler trick, try the above on an actual, physical device. Plug your device into your computer; if you’re on iOS 7 you may have to tell your device to “trust” the machine. Follow the steps above. Magic!

On Fellowship

I’m just about halfway through my OpenNews fellowship at ProPublica.

Plenty has been said about why developers should work in (and would enjoy) the newsroom environment. I had a hand in one piece and many others have chimed in with wonderful stories to that end. As someone who was considering applying for this fellowship last year, I didn’t need these beatitudes, so to speak. I’d been a developer in news for a couple years and I was already hooked on the satisfaction that came from building things that could make a difference in people’s lives — not to mention the interesting people, the pace of work, the variety, etc. (…if you’re a developer who hasn’t yet worked with journalism or civic hacking, then you should read all of those posts, though.)

So I’ve been thinking: what do I know about this now that could’ve enticed me even more a year ago? I very nearly didn’t apply, since I was putting it off and putting it off and putting it off…

Well, it’s a fellowship. Right… but being an OpenNews Fellow doesn’t feel so much like the cut-and-dry “…person appointed to a position granting a stipend and allowing for advanced study or research” or “member of a group having common characteristics.” (Hat-tip to Webster.) No, no, these don’t quite capture the sense of the team and community that you become a part of.

Aside from the easy joke (sorry, Annabel), I’ve heard us cheekily described as “the Wu-Tang of journalism” or a “journalism Justice League.” We obviously have fun with it and I’m not sure what other fellowships are like, but in this case I like to consider this fellowship to mean an “epic team of awesome,” or some (more eloquent) approximation. We’re not just here to do research and better ourselves and the newsrooms we work in — we’re here to foster and advance the overall journalism/code community.

We have backgrounds in programming, statistics, censorship research, cybersecurity, satellite communications, and we’re from nearly every corner of the earth. That range of backgrounds and skills has honestly made collaboration fun and at times surprising (in a good way).

We had a fellowship question-and-answer session in our weekly community call yesterday. Responding to a question about highs and lows in our experiences thusfar, a resounding “high point” for many of us was “any time the fellows get together.” (And this is no slight to the hosting news organizations and the day-to-day work we do and the adventures each of us has encountered. Hell, I work with an amazing and inspiring team that I once idolized and never fathomed I’d be working with.)

Another lesson (and high point) from my first few months: the community of civic-minded programmers and data people is huge now. It’s not just journalists anymore — there are people who want to help improve government, health, education, help make history more accessible, and on and on, outside of the guise of news media. There are communities of people who want to work together to do great things for the world, and these formerly-disparate communities are starting to mingle and work together in interesting. I’ve had the opportunity to visit the MIT Center for Civic Media a couple times and been blown away at the variety and scope of projects there — from free speech to gender to robots to accountability in developing nations. I’ve met the excellent people at the New York Public Library Labs team and had a chance to hack on historic geodata alongside geo developers that created the state of the art.

A lot of the best parts of the fellowship are about inspiration and turning that inspiration into something great. We fellows inspire one another, we inspire other people, and we in turn, are inspired by a lot of the people we get a chance to interact with through the course of our fellowship work.

Maybe what I should have said is less “fellowship as epic team” and more “fellowship as epic community.” (Or…)

The Knight-Mozilla OpenNews program exists at a great time in the journalism/civic code world. There’s an ecosystem of tools and datasets out there, and there’s a whole world of people that are in this space or interested in it. People are starting to get it that transparency and open tools and sharing and collaboration — especially when it comes to data about our governments and our environments — can foster real change and real impact. And it’s part of our mandate to get out there and work in the intersection of these communities — journalism, open source, civic impact — to make all of them better.

I mean, Source, the OpenNews site about “Journalism code and the people who make it”? Well whad’ya know, the site has appeal far beyond journalism — with posts (on crowdsourcing, hardware hacking, data visualization, and so on…) often landing on Hacker News and being discussed far from the walls of journalism institutions.

If working in this amazing community, with these amazing people sounds like something you want to take a part of, you should apply to become an OpenNews fellow. You should join the community. Hop into IRC. Listen in on our community calls. Find a civic-oriented hackday. Build some tools and share them with the world. Figure out how to get an obscure dataset into something usable — and share that knowledge so the next person can do it, too. Get involved and talk to people in the community — you’ll be amazed at how many in the community are not only brilliant developers/journalists/evangelists, but they’re wonderful and nice people, too.

There’s a whole community of people doing epic things to make the world a better place, and you can — and should! — be a part of it, too. It sounds daunting, sure, but the amount of inspiration and support you can find from colleagues and neighbors is equally great.

And seriously, if you’re a developer that wants to work on amazing things and get inspired and work with inspiring people — don’t be afraid to apply for the fellowship. I’m glad I wasn’t.

I’ve been a terrible blogger lately, but I’m going to be catching up on writing very soon. (Yes, I always say that.) I’ve got about as many “blog posts yet to write” as I have projects — and I’ve got an interesting plethora of those as of late.

…Actually, let’s talk about some of those for a minute. I plan on discussing a few of these at length over the next couple weeks (as big news is on the horizon for some), but here’s a collection of some things I’ve been working on since I last wrote:

I’ve probably forgotten something in there… But you’ll hear back from me soon enough.

(Are you a developer-ish person? Do these projects sound interesting to you? You should check out the Knight-Mozilla OpenNews Project, get involved with the gang, and apply to be a 2014 fellow. Seriously, check it out.)

I started blogging here about ten years ago, when I was still a kid, and when hand-editing a handful of HTML pages was an acceptable way to run a blog. (And no "permalinks"/"detail pages"!) LiveJournal was still invite-only. The first public release of Wordpress wouldn’t come happen for another six months. (And you’d have to host it yourself: the service wouldn’t happen for another three years.)

By “blogging here,” I mean here on this Ship of Theseus which was once straight hand-crafted HTML and then an AvidGamers CMS hack and then plain HTML again and then a pMachine CMS blog and then Wordpress and then a hand-crafted Django app. It has gone through enough thematic phases and has outlasted most Xanga blogs and MySpace profiles. But I still consider it one and the same as it’s only ever existed in one place at any given time (though the place has varied).

I’m sometimes called out for being too nostalgic, a trait which probably explains how intact the majority of the archive remains. There are some non-public posts in the mix, but I have generally dragged the bulk of it along publicly — including even the awkward, rambling, philosophizing teenager.

Now, a lot of that content is embarrassing (and I personally haven’t looked through it and re-vetted it in years), but I think I’ve stubbornly dragged it along as some sort of awkward personal benchmark: some form of this has existed for nearly half of the entire lifetime of the World Wide Web (circa 1991-92).

(To be fair to that count, this website is not a direct continuation of my first attempt, created in 1998. That <blink>-ing, <marquee>-scrolling, MIDI-playing, child-built website was a different beast altogether.)

Now, I’d say “I’ll get back into the habit of brain dumping here,” but I’ve never habitually done that (save for a few months maybe eight or nine years ago). But there’s big news to talk about, and hopefully I’ll get to talking about that — and other fun things — soon.

Here’s to another ten years and outlasting a few more hosting providers/blog sites/social networks.

django-medusa: Rendering Django sites as static HTML

If you’ve ever poked around on this blog, you may have noticed the colophon which mentions very briefly:

Unlike most Django sites, this is compiled into static HTML pages by django-medusa, a Django app I'm currently building.

This little tool has been open-source since I deployed to the new version of this website, maybe nine months ago, but I hadn’t really done anything with it or mentioned it much anywhere. It powers the several hundred pages on this site and turns them into static HTML — which is then hosted in S3. (Details below.)

(The only other time I’ve mentioned this project publicly was in response to django-bakery, a tool that the L.A. Times Data Desk uses to process some data projects into static pages. Clearly, this is an interesting idea to some people.)

tl;dr for Django pros: Test out the tutorial “hello world” and see the README. Come back if you want the more detailed narrative breakdown of the app (and how the app powers this blog).

The app basically auto-finds “renderer” definitions for your apps and then provides a Django management command that builds the static rendition of the website (with the output directed based on some settings).

Renderers live in files that are auto-discovered — like and, it’s auto-discovered as long as the app is listed in INSTALLED_APPS. This basically defines a class that defines a get_paths instance method that returns a list/tuple of absolute URLs, representing all the URLs that should be converted to static files. Renderers are set up like this so that, on an app-by-app basis (or even varying within an app), you can dynamically generate all the possible URLs that exist in your site.

Here’s a couple renderer definitions that actually power part of this site.

The specific URL names and model bits aren’t important: basically, you’ll notice that the example BlogHomeRenderer in my example generates the entire URL structure for /blog/* by querying for all live blog posts and then using Django’s URL reversing methods to figure out all the paths that could possibly be built. (That file in particular uses sets instead of lists/tuples, so that it can just blindly generate all the URLs and have duplicates ignored. It casts the set to a list upon returning.)

The process that actually generates the output simply uses (or abuses) Django’s internal testclient to request each URL and store the resulting data (and mimetype/other metadata, if using the right backend — I’ll touch on this more, below). I believe that this paradigm provides the most flexibility regarding giving each app the ability to define it’s own outputs and it keeps app-and-view-building as Django-like as possible (i.e. you are still building within the urlconf and view system). It seems ghetto at first to rely on those internal HTTP testclient mechanisms, but I haven’t yet encountered any issues — the rendering command can even (optionally) parallelize the testclient crawl to achieve faster rendering.

The staticsitegen management command then renders the URL structure you’ve defined, into static files. There are currently three rendering classes:

  • a disk-based renderer which outputs the directory tree in HTML files, turning bare URLs into directories with an index.html (so /colophon/ would result in output_dir/colophon/index.html being generated)
  • an Amazon S3 renderer which uploads the files directly to an S3 bucket (overriding duplicates)
  • a Google App Engine renderer which uploads the files to a static GAE instance, similar to the S3 renderer’s behavior

The advantages of the latter two primarily deal with situations where non-HTML content is generated: if any of your views returns JSON, XML, or some other data format, then the S3/GAE renderers will attempt to store the generated files with that mimetype.

This blog basically runs on a local dev server that uses an SQLite database as storage. I use the S3 renderer for this, to cut out the filesystem middle-man. For static files, I use the built-in staticfiles app along with django-storages; the collectstatic command automatically uploads static resources to S3, the same way the medusa S3 renderer does.

I write everything on that local server, then use my staticsitegen command to upload the whole URL tree. (In the event I updated any template bits, every URL is re-generated and overwritten.) I then use collectstatic to sync my static (CSS/JS/etc.) files. (For CloudFront or EdgeCast CDNs, right around here is where I’d run a script to immediately invalidate some of the more recent URL roots so that blog index pages get refreshed faster.)

That’s basically it. This blog (and the Onion Browser site, which are simply implemented as direct_to_template views) have been running via this system for about nine months and it’s been solid. Not even the Onion Browser release rush (being featured on Hacker News, Reddit, Gizmodo, Lifehacker, etc. etc. etc.) affected the site in the slightest.

Despite the static nature of the underlying pages, I don’t lose the ability to have comments, stats, and other features: I had been running Disqus for comments for quite some time, and I still use Google Analytics for analytics. (I recently disabled comments on this site; mainly out of apathy and lack of use than any philosophical stance.)

The README has a pretty good technical overview that goes into more code detail than the above paragraphs. You might want to start with this basic “hello world” tutorial first, though — it’ll get your feet wet and demonstrate the ease at which you can convert a (simple) Django project to become a statically-generated Django project, by simply adding a renderer definition and some settings.

I’m planning to clean the code a bit and tidy up (and add to) the documentation in the next couple weeks, but I figured I’d been sitting on this long enough.

You can discuss this on Hacker News. Feel free to bug me via e-mail or follow me on Twitter, too.