Archive for the ‘Programming’ Category

Fun, Frustration, Curiosity, Murderous Rage – mimic

Monday, January 15th, 2018

mimic

From the webpage:


There are many more characters in the Unicode character set that look, to some extent or another, like others – homoglyphs. Mimic substitutes common ASCII characters for obscure homoglyphs.

Fun games to play with mimic:

  • Pipe some source code through and see if you can find all of the problems
  • Pipe someone else’s source code through without telling them
  • Be fired, and then killed

I can attest to the murderous rage from experience. There was a browser-based SGML parser that would barf on the presence of an extra whitespace (space I think) in the SGML declaration. One file worked, another with the “same” declaration did not.

Only by printing and comparing the files (this was on Windoze machines) was the errant space discovered.

Enjoy!

Learn to Write Command Line Utilities in R

Thursday, December 21st, 2017

Learn to Write Command Line Utilities in R by Mark Sellors.

From the post:

Do you know some R? Have you ever wanted to write your own command line utilities, but didn’t know where to start? Do you like Harry Potter?

If the answer to these questions is “Yes!”, then you’ve come to the right place. If the answer is “No”, but you have some free time, stick around anyway, it might be fun!

Sellors invokes the tradition of *nix command line tools saying: “The thing that most [command line] tools have in common is that they do a small number of things really well.”

The question to you is: What small things do you want to do really well?

Practicing Vulnerability Hunting in Programming Languages for Music

Tuesday, December 19th, 2017

If you watched Natalie Silvanovich‘s presentation on mining the JavaScript standard for vulnerabilities, the tweet from Computer Science @CompSciFact pointing to Programming Languages Used for Music must have you drooling like one of Pavlov‘s dogs.

I count one hundred and forty-seven (147) languages, of varying degrees of popularity, none of which has gotten the security review of ECMA-262. (Michael Aranda wades through terminology/naming issues for ECMAScript vs. JavaScript at: What’s the difference between JavaScript and ECMAScript?.)

Good hunting!

A Little Story About the `yes` Unix Command

Tuesday, December 12th, 2017

A Little Story About the `yes` Unix Command by Matthais Endler.

From the post:

What’s the simplest Unix command you know?

There’s echo, which prints a string to stdout and true, which always terminates with an exit code of 0.

Among the rows of simple Unix commands, there’s also yes. If you run it without arguments, you get an infinite stream of y’s, separated by a newline:

Ever installed a program, which required you to type “y” and hit enter to keep going? yes to the rescue!

Endler sets out to re-implement the yes command in Rust.

Why re-implement Unix tools?

The trivial program yes turns out not to be so trivial after all. It uses output buffering and memory alignment to improve performance. Re-implementing Unix tools is fun and makes me appreciate the nifty tricks, which make our computers fast.

Endler’s story is unlikely to replace any of your holiday favorites but unlike those, it has the potential to make you a better programmer.

Releasing Failed Code to Distract from Accountability

Sunday, December 10th, 2017

Dutch government publishes large project as Free Software by
Carmen Bianca Bakker.

From the post:

The Dutch Ministry of the Interior and Kingdom Relations released the source code and documentation of Basisregistratie Personen (BRP), a 100M€ IT system that registers information about inhabitants within the Netherlands. This comes as a great success for Public Code, and the FSFE applauds the Dutch government’s shift to Free Software.

Operation BRP is an IT project by the Dutch government that has been in the works since 2004. It has cost Dutch taxpayers upwards of 100 million Euros and has endured three failed attempts at revival, without anything to show for it. From the outside, it was unclear what exactly was costing taxpayers so much money with very little information to go on. After the plug had been pulled from the project earlier this year in July, the former interior minister agreed to publish the source code under pressure of Parliament, to offer transparency about the failed project. Secretary of state Knops has now gone beyond that promise and released the source code as Free Software (a.k.a. Open Source Software) to the public.

In 2013, when the first smoke signals showed, the former interior minister initially wanted to address concerns about the project by providing limited parts of the source code to a limited amount of people under certain restrictive conditions. The ministry has since made a complete about-face, releasing a snapshot of the (allegedly) full source code and documentation under the terms of the GNU Affero General Public License, with the development history soon to follow.

As far as the “…complete about-face…,” the American expression is: “You’ve been had.

Be appearing to agonize over the release of the source code, the “former interior minister” has made it appear the public has won a great victory for transparency.

Actually not.

Does the “transparency” offered by the source code show who authorized the expenditure of each part of the 100M€ total and who was paid that 100M€? Does source code “transparency” disclose project management decisions and who, in terms of government officials, approved those project decisions. For that matter, does source code “transparency” disclose discussions of project choices at all and who was present at those discussions?

It’s not hard to see that source code “transparency” is a deliberate failure on the part of the Dutch Ministry of the Interior and Kingdom Relations to be transparent. It has withheld, quite deliberately, any information that would enable Dutch citizens, programmers or otherwise, to have informed opinions about the failure of this project. Or to hold any accountable for its failure.

This may be:

…an unprecedented move of transparency by the Dutch government….

but only if the Dutch government is a black hole in terms of meaningful accountability for its software projects.

Which appears to be the case.

PS: Assuming Dutch citizens can pry project documentation out of the secretive Dutch Ministry of the Interior and Kingdom Relations, I know some Dutch topic mappers could assist with establishing transparency. If that’s what you want.

The Top-100 rated Devoxx Belgium 2017 talks (or the full 207)

Thursday, December 7th, 2017

The Top-100 rated Devoxx Belgium 2017 talks

The top-100 list has Devoxx Belgium 2017 talks sorted in voting order, with hyperlinks to the top 50.

If you are looking for more comprehensive coverage of Devoxx Belgium 2017, try the Devoxx Belgium 2017 YouTube Playlist, with 207 videos!

Kudos to Devoxx for putting conference content online to spread the word about technology.

So You Want to be a WIZARD [Spoiler Alert: It Requires Work]

Monday, November 20th, 2017

So You Want to be a WIZARD by Julia Evans.

I avoid using terms like inspirational, transforming, etc. because it is so rare that software, projects, presentations merit merit those terms.

Today I am making an exception to that rule to say:

So You Want to be a Wizard by Julia Evans can transform your work in computer science.

Notice the use of “can” in that sentence. No guarantees because unlike many promised solutions, Julia says up front that hard work is required to use her suggestions successfully.

That’s right. If these methods don’t work for you it will be because you did not apply them. (full stop)

No guarantees you will get praise, promotions, recognition, etc., as a result of using Julia’s techniques, but you will be a wizard none the less.

One consolation is that wizards rarely notice back-biters, office sycophants, and a range of other toxic co-workers. They are too busy preparing themselves to answer the next technical issue that requires a wizard.

10 Papers Every Developer Should Read (At Least Twice) [With Hyperlinks]

Thursday, November 16th, 2017

10 Papers Every Developer Should Read (At Least Twice) by Michael Feathers

Feathers omits hyperlinks for the 10 papers every developer should read, at least twice.

Hyperlinks eliminate searches by every reader, saving them time and load on their favorite search engine, not to mention providing access more quickly. Feathers’ list with hyperlinks follows.

Most are easy to read but some are rough going – they drop off into math after the first few pages. Take the math to tolerance and then move on. The ideas are the important thing.

See Feather’s post for his comments on each paper.

Even a shallow web composed of hyperlinks is better than no web at all.

Scipy Lecture Notes

Sunday, November 12th, 2017

Scipy Lecture Notes edited by Gaël Varoquaux, Emmanuelle Gouillart, Olav Vahtras.

From the webpage:

Tutorials on the scientific Python ecosystem: a quick introduction to central tools and techniques. The different chapters each correspond to a 1 to 2 hours course with increasing level of expertise, from beginner to expert.

In PDF format, some six-hundred and fifty-seven pages of top quality material on Scipy.

In addition to the main editors, there are fourteen chapter editors and seventy-three contributors.

Good documentation needs maintenance so if you improvements or examples to offer, perhaps your name will appear here in the not too distant future.

Enjoy!

Introduction To ARM Assembly Basics [The Weakest Link?]

Friday, November 10th, 2017

Introduction To ARM Assembly Basics

The latest security fails by Intel and Microsoft capture media and blog headlines but ARM devices are more numerous.

ARM devices, like a Windows server in an unlocked closet, may be the weakest link in your next target.

From the webpage:

Welcome to this tutorial series on ARM assembly basics. This is the preparation for the followup tutorial series on ARM exploit development. Before we can dive into creating ARM shellcode and build ROP chains, we need to cover some ARM Assembly basics first.

The following topics will be covered step by step:

ARM Assembly Basics Tutorial Series:
Part 1: Introduction to ARM Assembly
Part 2: Data Types Registers
Part 3: ARM Instruction Set
Part 4: Memory Instructions: Loading and Storing Data
Part 5: Load and Store Multiple
Part 6: Conditional Execution and Branching
Part 7: Stack and Functions

To follow along with the examples, you will need an ARM based lab environment. If you don’t have an ARM device (like Raspberry Pi), you can set up your own lab environment in a Virtual Machine using QEMU and the Raspberry Pi distro by following this tutorial. If you are not familiar with basic debugging with GDB, you can get the basics in this tutorial. In this tutorial, the focus will be on ARM 32-bit, and the examples are compiled on an ARMv6.

Why ARM?

This tutorial is generally for people who want to learn the basics of ARM assembly. Especially for those of you who are interested in exploit writing on the ARM platform. You might have already noticed that ARM processors are everywhere around you. When I look around me, I can count far more devices that feature an ARM processor in my house than Intel processors. This includes phones, routers, and not to forget the IoT devices that seem to explode in sales these days. That said, the ARM processor has become one of the most widespread CPU cores in the world. Which brings us to the fact that like PCs, IoT devices are susceptible to improper input validation abuse such as buffer overflows. Given the widespread usage of ARM based devices and the potential for misuse, attacks on these devices have become much more common.

Yet, we have more experts specialized in x86 security research than we have for ARM, although ARM assembly language is perhaps the easiest assembly language in widespread use. So, why aren’t more people focusing on ARM? Perhaps because there are more learning resources out there covering exploitation on Intel than there are for ARM. Just think about the great tutorials on Intel x86 Exploit writing by Fuzzy Security or the Corelan Team – Guidelines like these help people interested in this specific area to get practical knowledge and the inspiration to learn beyond what is covered in those tutorials. If you are interested in x86 exploit writing, the Corelan and Fuzzysec tutorials are your perfect starting point. In this tutorial series here, we will focus on assembly basics and exploit writing on ARM.

Don’t forget to follow Azeria on Twitter, or her RSS Feed.

Enjoy!

PS: She recently posted an really cool cheatsheet: Assembly Basics Cheatsheet. I’m going to use it to lobby (myself) for a pair of 32″ monitors so I can enlarge it on one screen and have a non-scrolling display. (Suggestions on the monitors?)

Flight rules for git – How to Distinguish Between Astronauts and Programmers

Thursday, November 9th, 2017

Flight rules for git by Kate Hudson.

From the post:

What are “flight rules”?

A guide for astronauts (now, programmers using git) about what to do when things go wrong.

Flight Rules are the hard-earned body of knowledge recorded in manuals that list, step-by-step, what to do if X occurs, and why. Essentially, they are extremely detailed, scenario-specific standard operating procedures. […]

NASA has been capturing our missteps, disasters and solutions since the early 1960s, when Mercury-era ground teams first started gathering “lessons learned” into a compendium that now lists thousands of problematic situations, from engine failure to busted hatch handles to computer glitches, and their solutions.

— Chris Hadfield, An Astronaut’s Guide to Life.

Hudson devises an easy test to distinguish between astronauts and programmers:

Astronauts – missteps, disasters and solutions are written down.

Programmers – missteps, disasters and solutions are programmer/sysadmin lore.

With Usenet and Stackover, you can argue improvement by programmers but it’s hardly been systematic. Even so it depends on a “good” query returning few enough “hits” to be useful.

Hudson is capturing “flight rules” for git.

Act like an astronaut and write down your missteps, disasters and solutions.

NASA made it to the moon and beyond by writing things down.

Who knows?

Writing down software missteps, disasters and solutions may help render all systems transparent, willingly or not.

SciPy 1.0.0! [Awaiting Your Commands]

Thursday, October 26th, 2017

SciPy 1.0.0

From the webpage:

We are extremely pleased to announce the release of SciPy 1.0, 16 years after version 0.1 saw the light of day. It has been a long, productive journey to get here, and we anticipate many more exciting new features and releases in the future.

Why 1.0 now?

A version number should reflect the maturity of a project – and SciPy was a mature and stable library that is heavily used in production settings for a long time already. From that perspective, the 1.0 version number is long overdue.

Some key project goals, both technical (e.g. Windows wheels and continuous integration) and organisational (a governance structure, code of conduct and a roadmap), have been achieved recently.

Many of us are a bit perfectionist, and therefore are reluctant to call something “1.0” because it may imply that it’s “finished” or “we are 100% happy with it”. This is normal for many open source projects, however that doesn’t make it right. We acknowledge to ourselves that it’s not perfect, and there are some dusty corners left (that will probably always be the case). Despite that, SciPy is extremely useful to its users, on average has high quality code and documentation, and gives the stability and backwards compatibility guarantees that a 1.0 label imply.

In case your hands are trembling too much to type in the URLs:

SciPy.org

SciPy Cookbook

Scipy 1.0.0 Reference Guide, [HTML+zip], [PDF]

Like most tools, it isn’t weaponized until you apply it to data.

Enjoy!

PS: If you want to get ahead of a co-worker, give them this URL: http://planet.scipy.org/. Don’t look, it’s a blog feed for SciPy. Sorry, you looked didn’t you?

How To Be A Wizard Programmer – Julia Evans @b0rk

Monday, October 9th, 2017

See at full scale.

Criticism: Julia does miss one important step!

Follow: Julia Evans @b0rk

😉

Building Data Science with JS – Lifting the Curtain on Game Reviews

Saturday, October 7th, 2017

Building Data Science with JS by Tim Ermilov.

Three videos thus far:

Building Data Science with JS – Part 1 – Introduction

Building Data Science with JS – Part 2 – Microservices

Building Data Science with JS – Part 3 – RabbitMQ and OpenCritic microservice

Tim starts with the observation that the percentage of users assigning a score to a game isn’t very helpful. It tells you nothing about the content of the game and/or the person rating it.

In subject identity terms, each level, mighty, strong, weak, fair, collapses information about the game and a particular reviewer into a single summary subject. OpenCritic then displays the percent of reviewers who are represented by that summary subject.

The problem with the summary subject is that one critic may have down rated the game for poor content, another for sexism and still another for bad graphics. But a user only knows for reasons unknown, a critic whose past behavior is unknown, evaluated unknown content and assigned it a rating.

A user could read all the reviews, study the history of each reviewer, along with the other movies they have evaluated, but Ermilov proposes a more efficient means to peak behind the curtain of game ratings. (part 1)

In part 2, Ermilov designs a microservice based application to extract, process and display game reviews.

If you thought the first two parts were slow, you should enjoy Part 3. 😉 Ermilov speeds through a number of resources, documents, JS libraries, not to mention his source code for the project. You are likely to hit pause during this video.

Some links you will find helpful for Part 3:

AMQP 0-9-1 library and client for Node.JS – Channel-oriented API reference

AMQP 0-9-1 library and client for Node.JS (Github)

https://github.com/BuildingXwithJS

https://github.com/BuildingXwithJS/building-data-science-with-js

Microwork – simple creation of distributed scalable microservices in node.js with RabbitMQ (simplifies use of AMQP)

node-unfluff – Automatically extract body content (and other cool stuff) from an html document

OpenCritic

RabbitMQ. (Recommends looking at the RabbitMQ tutorials.)

Exploratory Data Analysis of Tropical Storms in R

Tuesday, September 26th, 2017

Exploratory Data Analysis of Tropical Storms in R by Scott Stoltzman.

From the post:

The disastrous impact of recent hurricanes, Harvey and Irma, generated a large influx of data within the online community. I was curious about the history of hurricanes and tropical storms so I found a data set on data.world and started some basic Exploratory data analysis (EDA).

EDA is crucial to starting any project. Through EDA you can start to identify errors & inconsistencies in your data, find interesting patterns, see correlations and start to develop hypotheses to test. For most people, basic spreadsheets and charts are handy and provide a great place to start. They are an easy-to-use method to manipulate and visualize your data quickly. Data scientists may cringe at the idea of using a graphical user interface (GUI) to kick-off the EDA process but those tools are very effective and efficient when used properly. However, if you’re reading this, you’re probably trying to take EDA to the next level. The best way to learn is to get your hands dirty, let’s get started.

The original source of the data was can be found at DHS.gov.

Great walk through on exploratory data analysis.

Everyone talks about the weather but did you know there is a forty (40) year climate lag between cause and effect?

The human impact on the environment today, won’t be felt for another forty (40) years.

Can to predict the impact of a hurricane in 2057?

Some other data/analysis resources on hurricanes, Climate Prediction Center, Hurricane Forecast Computer Models, National Hurricane Center.

PS: Is a Category 6 Hurricane Possible? by Brian Donegan is an interesting discussion on going beyond category 5 for hurricanes. For reference on speeds, see: Fujita Scale (tornadoes).

MIT License Wins Converts (some anyway)

Friday, September 22nd, 2017

Relicensing React, Jest, Flow, and Immutable.js by Adam Wolff.

From the post:

Next week, we are going to relicense our open source projects React, Jest, Flow, and Immutable.js under the MIT license. We’re relicensing these projects because React is the foundation of a broad ecosystem of open source software for the web, and we don’t want to hold back forward progress for nontechnical reasons.

This decision comes after several weeks of disappointment and uncertainty for our community. Although we still believe our BSD + Patents license provides some benefits to users of our projects, we acknowledge that we failed to decisively convince this community.

In the wake of uncertainty about our license, we know that many teams went through the process of selecting an alternative library to React. We’re sorry for the churn. We don’t expect to win these teams back by making this change, but we do want to leave the door open. Friendly cooperation and competition in this space pushes us all forward, and we want to participate fully.

This shift naturally raises questions about the rest of Facebook’s open source projects. Many of our popular projects will keep the BSD + Patents license for now. We’re evaluating those projects’ licenses too, but each project is different and alternative licensing options will depend on a variety of factors.

We’ll include the license updates with React 16’s release next week. We’ve been working on React 16 for over a year, and we’ve completely rewritten its internals in order to unlock powerful features that will benefit everyone building user interfaces at scale. We’ll share more soon about how we rewrote React, and we hope that our work will inspire developers everywhere, whether they use React or not. We’re looking forward to putting this license discussion behind us and getting back to what we care about most: shipping great products.

Since I bang on about Facebook‘s 24×7 censorship and shaping of your worldview, it’s only fair to mention when they make a good choice.

It in no way excuses or justifies their ongoing offenses against the public but it’s some evidence that decent people remain employed at Facebook.

With any luck, the decent insiders will wrest control of Facebook away from its government toadies and collaborators.

RStartHere

Monday, September 18th, 2017

RStartHere by Garrett Grolemund.

R packages organized by their role in data science:

This is very cool! Use and share!

@rstudio Cheatsheets Now B&W Printer Friendly

Saturday, September 9th, 2017

Mara Averick, @dataandme, tweets:

All the @rstudio Cheatsheets have been B&W printer-friendlier-ized

It’s a small thing but appreciated when documentation is B&W friendly.

PS: The @rstudio cheatsheets are also good examples layout and clarity.

The International Conference on Functional Programming – 2017

Tuesday, September 5th, 2017

The International Conference on Functional Programming – 2017 – Papers

If you are on the Gulf or East coast of the United States, take this opportunity to download papers to read following land fall of Irma.

You may not have Internet service but if you have printed several papers out as emergency preparedness, you won’t be at a loss for reading materials.

I’ve been in the impact zone of several hurricanes and while reading materials don’t make repairs go any faster, they do help pass the time.

Reinventing Wheels with No Wheel Experience

Friday, June 30th, 2017

Rob Graham, @ErrataRob, captured an essential truth when he tweeted:

Wheel re-invention is inherent every new programming language, every new library, and no doubt, nearly every new program.

How much “wheel experience” every programmer has across the breath of software vulnerabilities?

Hard to imagine meaningful numbers on the “wheel experience” of programmers in general but vulnerability reports make it clear either “wheel experience” is lacking or the lesson didn’t stick. Your call.

Vulnerabilities may occur in any release so standard practice is to check every release, however small. Have your results independently verified by trusted others.

PS: For the details on systemd, see: Sergey Bratus and the systemd thread.

You Are Not Google (Blasphemy I Know, But He Said It, Not Me)

Thursday, June 8th, 2017

You Are Not Google by Ozan Onay.

From the post:

Software engineers go crazy for the most ridiculous things. We like to think that we’re hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person’s Hacker News comment to another’s blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.

This is not how rational people make decisions, but it is how software engineers decide to use MapReduce.

Spoiler: Onay will also say you are not Amazon or LinkedIn.

Just so you know and can prepare for the ego shock.

Great read that invokes Poyla’s First Principle:


Understand the Problem

This seems so obvious that it is often not even mentioned, yet students are often stymied in their efforts to solve problems simply because they don’t understand it fully, or even in part. Polya taught teachers to ask students questions such as:

  • Do you understand all the words used in stating the problem?
  • What are you asked to find or show?
  • Can you restate the problem in your own words?
  • Can you think of a picture or a diagram that might help you understand the problem?
  • Is there enough information to enable you to find a solution?

Onay coins a mnemonic for you to apply and points to additional reading.

Enjoy!

PS: Caution: Understanding a problem can cast doubt on otherwise successful proposals for funding. Your call.

Copy-n-Paste Security Alert!

Wednesday, June 7th, 2017

Security: The Dangers Of Copying And Pasting R Code.

From the post:

Most of the time when we stumble across a code snippet online, we often blindly copy and paste it into the R console. I suspect almost everyone does this. After all, what’s the harm?

The post illustrates how innocent appearing R code can conceal unhappy surprises!

Concealment isn’t limited to R code.

Any CSS controlled display is capable of concealing code for you to copy-n-paste into a console, terminal window, script or program.

Endless possibilities for HTML pages/emails with code + a “little something extra.”

What are your copy-n-paste practices?

C Reference Manual (D.M. Richie, 1974)

Tuesday, May 23rd, 2017

C Reference Manual (D.M. Richie, 1974)

I mention the C Reference Manual, now forty-three (43) years old, as encouragement to write good documentation.

It may have a longer life than you ever expected!

For example, in 1974 Richie writes:

2.2 Identifier (Names)

An identifier is a sequence of letters and digits: the first character must be alphabetic.

Which we find replicated years later in ISO/IEC 8879 : 1986 (SGML):

4.198 name: A name token whose first character is a name start character.

4.201 name start character: A character that can begin a name: letters and others designated by the concrete syntax.

And in production [53]:


name start character =
LC Letter \
UC Letter \
LCNMSTRT \
UCNMSTRT

Where Figure 1 of 9.2.1 SGML Character defines LC Letter as a-z, UC Letter as A-Z, LCNMSTRT as (none), UCNMSTRT as (none), in the concrete syntax.

And in 1997, the letter vs. digit distinction, finds its way into Extensible Markup Language (XML) 1.0.


[4] NameChar ::= Letter | Digit | ‘.’ | ‘-‘ | ‘_’ | ‘:’ | CombiningChar | Extender
[5] Name ::= (Letter | ‘_’ | ‘:’) (NameChar)*

“Letter” is a link to a production referencing all the qualifying Unicode characters which is too long to include here.

What started off as an arbitrary choice, “alphabetic” characters as name start characters in 1974, is picked up some 12 years later (1986) in ISO/IEC 8879 (SGML), both of which were bound by a restricted character set.

When the opportunity came to abandon the letter versus digit distinction in name start characters (XML 1.0), the result is a larger character repertoire for name start characters, but digits continue as second-class citizens.

Can you point to an explanation why Richie preferred alphabetic characters over digits for name start characters?

ARM Releases Machine Readable Architecture Specification (Intel?)

Saturday, April 22nd, 2017

ARM Releases Machine Readable Architecture Specification by Alastair Reid.

From the post:

Today ARM released version 8.2 of the ARM v8-A processor specification in machine readable form. This specification describes almost all of the architecture: instructions, page table walks, taking interrupts, taking synchronous exceptions such as page faults, taking asynchronous exceptions such as bus faults, user mode, system mode, hypervisor mode, secure mode, debug mode. It details all the instruction formats and system register formats. The semantics is written in ARM’s ASL Specification Language so it is all executable and has been tested very thoroughly using the same architecture conformance tests that ARM uses to test its processors (See my paper “Trustworthy Specifications of ARM v8-A and v8-M System Level Architecture”.)

The specification is being released in three sets of XML files:

  • The System Register Specification consists of an XML file for each system register in the architecture. For each register, the XML details all the fields within the register, how to access the register and which privilege levels can access the register.
  • The AArch64 Specification consists of an XML file for each instruction in the 64-bit architecture. For each instruction, there is the encoding diagram for the instruction, ASL code for decoding the instruction, ASL code for executing the instruction and any supporting code needed to execute the instruction and the decode tree for finding the instruction corresponding to a given bit-pattern. This also contains the ASL code for the system architecture: page table walks, exceptions, debug, etc.
  • The AArch32 Specification is similar to the AArch64 specification: it contains encoding diagrams, decode trees, decode/execute ASL code and supporting ASL code.

Alastair provides starting points for use of this material by outlining his prior uses of the same.

Raises the question why an equivalent machine readable data set isn’t available for Intel® 64 and IA-32 Architectures? (PDF manuals)

The data is there, but not in a machine readable format.

Anyone know why Intel doesn’t provide the same convenience?

Build Your Own Text Editor (“make changes, see the results”)

Thursday, April 6th, 2017

Build Your Own Text Editor by Jeremy Ruten.

From the webpage:

Welcome! This is an instruction booklet that shows you how to build a text editor in C.

The text editor is antirez’s kilo, with some changes. It’s about 1000 lines of C in a single file with no dependencies, and it implements all the basic features you expect in a minimal editor, as well as syntax highlighting and a search feature.

This booklet walks you through building the editor in 184 steps. Each step, you’ll add, change, or remove a few lines of code. Most steps, you’ll be able to observe the changes you made by compiling and running the program immediately afterwards.

I explain each step along the way, sometimes in a lot of detail. Free free to skim or skip the prose, as the main point of this is that you are going to build a text editor from scratch! Anything you learn along the way is bonus, and there’s plenty to learn just from typing in the changes to the code and observing the results.

See the appendices for more information on the tutorial itself (including what to do if you get stuck, and where to get help).

If you’re ready to begin, then go to chapter 1!
… (emphasis in original)

I mention this tutorial because:

  • It’s an opportunity to see editor issues “from the other side.”
  • Practice reading and understanding C
  • I like the “make changes, see the results” approach

Of the three, the “make changes, see the results” approach is probably the most important.

Examples that “just work” are great and I look for them all the time. 😉

But imagine examples that take you down the false leads and traps, allowing you to observe the cryptic error messages from XQuery for example. You do work your way to a solution but are not given one out of the box.

“Cryptic” is probably overly generous with regard to XQuery error messages. Suggestions of a better one word term, usable in mixed company for them?

Eroding the Presumption of Innocence in USA

Saturday, April 1st, 2017

You may be laboring under the false impression that people charged with crimes in the USA are presumed innocence until proven guilty beyond a reasonable doubt in a court of law.

I regret to inform you that presumption is being eroded away.

Kevin Poulsen has a compelling read in FBI Arrests Hacker Who Hacked No One about the case of Taylor Huddleston was arraigned on March 31, 2017 in the Federal District Court for the Eastern District of Virginia, docket number: 1:2017 cr 34.

Kevin’s crime? He wrote a piece of software that has legitimate uses, such as sysadmins trouble shooting a user’s computer remotely. That tool was pirated by others and put to criminal use. Now the government wants to take his freedom and his home.

Compare Kevin’s post to the indictment, which I have uploaded for your reading pleasure. There is a serious disconnect between Poulsen’s post and the indictment, as the government makes much out of a lot of hand waving and very few specifics.

Taylor did obtain a Release on Personal Recognizance or Unsecured Bond, which makes you think the judge isn’t overly impressed with the government’s case.

I would have jumped at such a release as well but I find it disturbing, from a presumption of innocence perspective, that the judge also required:

My transcription:

No access to internet through any computer or other data capable device including smart phones

Remember that Taylor Huddleston is presumed innocence so how is that consistent with prohibiting him from a lawful activity, such as access to the internet?

Simple response: It’s not.

As I said, I would have jumped at the chance for a release on personal recognizance too. Judges are eroding the presumption of innocence with the promise of temporary freedom.

Wishing Huddleson the best of luck and that this erosion of the presumption of innocence won’t go unnoticed/unchallenged.

Notes to (NUS) Computer Science Freshmen…

Monday, March 13th, 2017

Notes to (NUS) Computer Science Freshmen, From The Future

From the intro:

Early into the AY12/13 academic year, Prof Tay Yong Chiang organized a supper for Computer Science freshmen at Tembusu College. The bunch of seniors who were gathered there put together a document for NUS computing freshmen. This is that document.

Feel free to create a pull request to edit or add to it, and share it with other freshmen you know.

There is one sad note:


The Art of Computer Programming (a review of everything in Computer Science; pretty much nobody, save Knuth, has finished reading this)

When you think about the amount of time Knuth has spent researching, writing and editing The Art of Computer Programming (TAOCP), it doesn’t sound unreasonable to expect others, a significant number of others, to have read it.

Any online reading groups focused on TAOCP?

Software Is Politics [Proudhon’s Response]

Sunday, February 19th, 2017

Software Is Politics by Richard Pope.

From the post:

If you work in software or design in 2016, you also work in politics. The inability of Facebook’s user interface, until recently, to distinguish between real and fake news is the most blatant example. But there are subtler examples all around us, from connected devices that threaten our privacy to ads targeting men for high-paying jobs.

Digital services wield power. They can’t be designed simply for ease of use—the goal at most companies and organizations. Digital services must be understandable, accountable, and trusted. It is now a commercial as well as a moral imperative.

DESIGN IS POLITICAL

Power and politics are not easy topics for many designers to chew on, but they’re foundational to my career. I worked for the U.K.’s Government Digital Service for five years, part of the team that delivered Gov.uk. I set up the labs team at Consumer Focus, the U.K.’s statutory consumer rights organization, building tools to empower consumers. In 2007, I cofounded the Rewired State series of hackdays that aimed to get developers and designers interested in making government better. I’ve also worked at various commercial startups including moo.com and ScraperWiki.

The last piece of work I did in government was on a conceptual framework for the idea of government as a platform. “Government as a platform” is the idea of treating government like a software stack to make it possible to build well-designed services for people. The work involved sketching some ideas out in code, not to try and solve them upfront, but to try and identify where some of the hard design problems were going to be. Things like: What might be required to enable an end-to-end commercial service for buying a house? Or what would it take for local authorities to be able to quickly spin up a new service for providing parking permits?

With this kind of thinking, you rapidly get into questions of power: What should the structure of government be? Should there be a minister responsible for online payment? Secretary of state for open standards? What does it do to people’s understanding of their government?

Which cuts to the heart of the problem in software design today: How do we build stuff that people can understand and trust, and is accountable when things go wrong? How do we design for recourse?
… (emphasis in original)

The flaw in Pope’s desire for applications are “…accountable, understandable, and trusted…” by all, is that it conceals the choosing of sides.

Or as Craig Gurian in Equally free to sleep under the bridge illustrates by quoting Anatole France:

“In its majestic equality, the law forbids rich and poor alike to sleep under bridges, beg in the streets and steal loaves of bread.”

Applications that are “…accountable, understandable, and trusted…” will have silently chosen sides just as the law does now.

Better to admit to and make explicit the choices of who serves and who eats in the design of applications. At least then disparities are not smothered by the pretense of equality.

Or as Proudhon would say:

What is equality before the law without equality of fortunes? A balance with false weights.

Speak not of “…accountable, understandable, and trusted…” applications in the abstract but for and against who?

Fundamentals of Functional Programming (email lessons)

Tuesday, February 14th, 2017

Learn the fundamentals of functional programming — for free, in your inbox by Preethi Kasireddy.

From the post:

If you’re a software developer, you’ve probably noticed a growing trend: software applications keep getting more complicated.

It falls on our shoulders as developers to build, test, maintain, and scale these complex systems. To do so, we have to create well-structured code that is easy to understand, write, debug, reuse, and maintain.

But actually writing programs like this requires much more than just practice and patience.

In my upcoming course, Learning Functional JavaScript the Right Way, I’ll teach you how to use functional programming to create well-structured code.

But before jumping into that course (and I hope you will!), there’s an important prerequisite: building a strong foundation in the underlying principles of functional programming.

So I’ve created a new free email course that will take you on a fun and exploratory journey into understanding some of these core principles.

Let’s take a look at what the email course will cover, so you can decide how it fits into your programming education.
…(emphasis in original)

I haven’t taken an email oriented course in quite some time so interested to see how this contrasts with video lectures, etc.

Enjoy!

A Data Driven Exploration of Kung Fu Films

Tuesday, January 24th, 2017

A Data Driven Exploration of Kung Fu Films by Jim Vallandingham.

From the post:

Recently, I’ve been a bit caught up in old Kung Fu movies. Shorting any technical explorations, I have instead been diving head-first into any and all Netflix accessible martial arts masterpieces from the 70’s and 80’s.

While I’ve definitely been enjoying the films, I realized recently that I had little context for the movies I was watching. I wondered if some films, like our latest favorite, Executioners from Shaolin, could be enjoyed even more, with better understanding of the context in which these films exist in the Kung Fu universe.

So, I began a data driven quest for truth and understanding (or at least a semi-interesting dataset to explore) of all Shaw Brothers Kung Fu movies ever made!

If you’re not familiar with the genre, here is a three-minute final fight collage from YouTube:

When I saw the title, I was hopeful that Jim had captured the choreography of the movies for comparison.

No such luck! 😉

That would be an extremely difficult and labor intensive task.

Just in case you are curious, there is a Dance Notation Bureau with extensive resources should you decide to capture one or more Kung Fu films in notation.

Or try Notation Reloaded: eXtensible Dance Scripting Notation by Matthew Gough.

A search using “xml dance notation” produces a number of interesting resources.