Posts Tagged ‘ SCDL

Encapsule Project Next Mocha/Chai Test Author(s)

If you look at my GitHub public commit logs you’ll notice that there’s been very little activity here in 2014. This is deceptive; I’ve joined a start-up in Seattle called Azuqua (http://azuqua.com) and am actively working on a major update to the service based, architecturally, on Object Namespace Manager, Software Circuit Description Language (onmd-scdl npm package), and some other for-now secret sauces (hint: graph theory).

onm npm package is stable (both in the browser via browserify) and on Node.js and I’m actively pushing bug fixes and small enhancements as I find/need them for production work. If you’re interested in using this package in your app and want more information drop me an e-mail. I am happy to answer questions and help as I’m able.

Contributing

I was talking with John-David Dalton (lodash author) after a recent SeattleJS meetup and he advised me to party on Mocha/Chai for my testing needs. So I dove in and think Mocha/Chai is really great. I’ve been writing a ton of tests for new code that derives from onm, but just don’t have the time to circle back and do the same for onm itself.

If you’re interested in learning about the onm lib, I mean really learning it, and are willing to help out writing Mocha/Chai tests please drop me a line.

September 2013 Project Update

Greetings. It’s been several months since my last post here and there’s a lot of new stuff to talk about.

In May I gave an hour-long demo and Q&A on the Encapsule Project Schema HTML 5 application at the Seattle Graph Database Meetup group. I felt like maybe half of the twenty or so people who came that evening understood the whole point of Soft Circuit Description Language (SCDL). Of that ten, perhaps two people were able to relate the concepts I discussed back to the rather boring Schema HTML 5 app demo I brought along to show.

Not a big win but I did get some good questions. The most interesting was if I was going to enable to programmatic creation of SCDL model and specification data or was simply going to provide the Schema SCDL Editor. I hadn’t previously considered programmatic generation of SCDL to be a high priority and was curious about what people would use such a facility for if it did exist.

Programmatic Generation of SCDL and Program Annotation

Why do we need a programmatic way to generate SCDL in addition to the Schema SCDL editor? The answer is fascinating; providing an API for manipulating SCDL programmatically makes it possible to write programs that write programs (i.e. a program synthesizer). Synthesis as you’ll recall involves the combination of two or more things to make something new. A SCDL program synthesizer specifically leverages SCDL as a domain specific annotation language.

For example a SCDL model set of re-usable data visualization component comprises a kit for building entire class of visualization applications (the class of all possible allowed combinations of the various SCDL models). A SCDL program synthesizer program could be written to glean salient information and features of an input data set(s) (e.g. via parsing), and use this information to generate a SCDL specification to visualize the data in terms of the available SCDL data visualization components.

In this example, what we’re annotating is how we want another program to visualize the input data given its structure, features, content… The resultant “annotation”, a SCDL specification, could then be executed to visualize the data, or serialized and passed along with the data to some other processing entity (e.g. to respond to a request).

To take a far less trivial example, the original question was posed by a gentleman whose job it is to figure out how to programmatically port thousands of legacy enterprise applications to the web. Plausibly I can imagine SCDL providing a framework for building up such transformation systems in a predictable way that scales. The viability of this approach of course depends on much infrastructure that doesn’t yet exist so currently this is theoretical. However, the entire exchange convinced me that I must provide a way to programmatically generate SCDL models and specifications.

In the course of this discussion it was mentioned that I’m rather blithely ignoring the semantic web. I countered that I’ve tried but just don’t get it and that perhaps by working through things my way I’ll understand. Or, be able to demonstrate a simpler way to affect the same ends using data flow graphs.

Schema HTML SCDL Editor and SCDL API

I’ve spent this entire year so far essentially trying to figure out how to write HTML 5 apps like Schema. It’s actually quite a difficult task to build an interactive data editing and visualization application when your data model is complicated. There’s a ton of runtime state and keeping the code clean defect free is extremely hard unless you’ve got some solid low-level infrastructure code.

As of the May demo I was still struggling with fundamental architecture questions about how I was going to structure the application and prevent myself from building something so sprawling that I would not longer be able to comprehend it.

Questions about support for a SCDL API got me thinking that the whatever solution I came up with had to both provide a nice and clean API but additionally support the following core scenarios for building interactive data editing / visualization apps in the browser:

  • The app data model must be declared at design-time and then used at runtime to tailor the specific functionality of generic routines to ensure that changes to the data model do not impose huge refactoring penalties.
  • All application data needs to be addressable in order to ensure a clean decoupling between the many routines that operate on it. Further the addressing schema must accommodate local hash URL routing as well as some future URI-based protocol.
  • It needs to be easy to segregate actual application runtime data from metadata.
  • It needs to be easy to write simple routines that operate on both app data and app metadata stores.
  • It must be trivial to save / restore all runtime app state in JSON format.

My summer has been consumed with designing a solution that meets all the above criteria and the result is a new Javscript library I call ONMjs.

ONMjs

ONM stands for Object Namespace Manager. The ‘js’ is tacked on the end to signify that this is a Javascript implementation of the library (others are certainly possible but for now, just Javascript).

The currently published version of Schema is essentially a complete re-write of the demo I gave in May that leverages the new ONMjs library to manage all of its application state.

What you see in the browser, in the little window panes, are actually little plug-ins that observe and react to changes in runtime application data via a sophisticated implementation of the Observer Pattern. Some even change the data and ONMjs takes care of letting everyone else know.

Over the next week I’ll be forking the ONMjs library into its own repo on GitHub and plan a series of small example apps and blog posts to explain what it is and how to use it in your own applications.

ONMjs is completely generic, is dependency-free Javascript, and is intended to be used in conjunction with libraries such as D3js and Knockoutjs (or with any number of other libraries).

If you’re working on this sort of web app ONMjs wants to be your friend. Stay tuned.

May 30th, 2013 Talk at Seattle Graph Database Meetup

Graph Database Seattle

Graph Database Seattle

I’ll be doing a talk about the Encapsule Project and the Schema HTML 5 application at the Seattle Graph Database Meetup group on Thursday, May 30th, 2013 in Seattle.

Event link: http://seattle.meetup.neo4j.org/events/117587312/

Disney’s 10th Floor World & Land, 925 4th Ave N Seattle,98104, Seattle, WA (map)

Thanks to Saikat Kanjilal and the other organizers of Graph Database Seattle for organizing these meetups and agreeing to let me speak!

Schema v0.869 Screenshot

I am currently working on something I call the “navigator” – essentially a Javascript library (compiled from Coffeescript actually)  that accepts as input a Javascript “layout” object declaration.

demo: stable demo link • latest development build

The layout is parsed to create a “tree-view” type thing. However, this isn’t an ordinary tree-view control. Not even close. The ‘navigator’ library allows each node in the tree to be annotated with additional metadata that conveys to the navigator that the menu item should semantically model a concept. “Concepts” are whatever you can declare and insert into the metadata.

The “concept” I’m interested in at the moment is modeling the structure of the SCDL data object model on top of JSON with enough precision to generate the runtime code I need to edit and manage the data objects in my model at the semantic level of SCDL, not JSON.

Starting with the simplest of Javascript objects that defines the menu names in a treeview, I’ve applied the idea of annotating each menu item with additional metadata fed into a set of object factories to allow menu items to be “conceptually bound” to nodes in a JSON object described in metadata.

A tree-view with a single root can be thought of as the root object of a JSON deserialization. Similarly, the children of the root can be thought of as sub-objects, or sub-arrays. Each menu item is annotated in metadata to set it’s associated JSON object type. Additionally, information in the metadata is parsed to determine SCDL-level schema (higher-order than JSON).

The screenshot below is of build 0.869 with the “Advanced” view selected.

Encapsule Project Schema 0.869 (test build)

Encapsule Project Schema 0.869 (test build)

It’s not time-efficient to explain further right now but the level of power the ‘navigator’ gives me is incredible… Navigator allows me to map pretty much any object model to JSON (doesn’t have to be SCDL). You can declare “navigator layout” objects (Javascript objects) for different problem domains (i.e. completely unrelated sets of objects that comprise some sort of “domain specific modeling schema”) and use them in the same application in different instances of navigator. Or, bind a single layout to multiple object models…

v1 Schema will very likely rely very heavily on ‘navigator’ as the core of the application.

Framing the SCDL Data Model in HTML 5

Today’s Schema build is the best SCDL reference I’ve written to date: http://www.chrisrussell.net/html5/schema-3/schema.html

Today marks my fortieth consecutive day of heads-down coding on Schema – a single page HTML 5 application for designing systems in Soft Circuit Description Language (SCDL – pronounced “scuddle” as in to make haste).

About half of this time has been consumed learning tricky details of other people’s amazing HTML 5 libraries (e.g. Knockout.js is really great).

The other half has been devoted to struggling to codify the SCDL object model (serialized to JSON and used to synthesize runtime code).

I’ve strongly resisted the urge to write about SCDL here as I’ve explained the concept countless times to some very smart people with little success to-date. I’m resigned to the fact that nobody is going to “get it” until they can see it. That is see and edit the graph models that underpin SCDL using browser-based SVG visualizations.

Over the past week I basically re-wrote the entire SCDL data model for Schema because it had just gotten out of control (entire data and view model in a single Coffeescript file). That single file is now twenty and I’m glad I did this now although I didn’t get a lot of sleep this week :-)

Tuesday I reached parity with where I was when I started refactoring and today the SCDL data model is almost completely framed in Coffeescript. It’s not yet functional (e.g. there are lots of methods missing) but the overall structure has been captured in classes.

Today’s development build of Schema is interesting. Essentially it’s a Knockout.js view bound to the SCDL data model with buttons that allow to instantiate objects. As changes are made, the SCDL catalogue JSON is updated dynamically.

If you’re intrigued by SCDL, check out today’s build. http://www.chrisrussell.net/html5/schema-3/schema.html

Note that I’ve cached today’s build on my own website for future reference. Once the SVG visualizations are online, most people will not care about this detail (that’s the whole point after all). But today’s build is perhaps the best SCDL reference I’ve written to-date so I’m sharing with those of you interested in this early work.

 

 

It’s quiet in here :)

I’m reserving this blog for talking about building software with circuit models. But in order to make it possible to efficiently even write about the subject I need pictures. Even better, I need a way to quickly make pictures. Even better, the pictures should be interactive, editable, savable, shareable…

This is the forthcoming “Schema” app for the browser I’ve mentioned briefly here. I’ve been busy: as of this writing encapsule/schema repository on GitHub is 66-commits old young and I’m on an unbroken 25-day commit spree. If you’re interested in my adventures in HTML5 open source development you can read the day-to-day here: blog.chrisrussell.net.

Early Schema SCDL Editor Demo

I’ve deployed a snapshot build of work in progress on Encapsule Schema – a single-page HTML5 application for creating, visualizing, and editing JSON-encoded Soft Circuit Description Language (SCDL) system models.

Demo snapshot build: http://www.chrisrussell.net/html5/schema-1/

Although primarily a demo of single-page, cached HTML5 application development at this point. You can actually explore the SCDL data model a bit but adding and removing SCDL entities from the forms embedded in the app’s main page.

SCDL Catalogue JSON

Schema Editor Produces SCDL in JSON

As you add/remove SCDL entities from a SCDL catalogue, you’ll note that the JSON representation of SCDL catalogue is updated live.

This is not even pre-alpha at this point. But a good start :-)

If you’re interested in the HTML5 client development that’s going into the Schema editor, you can follow along on my personal blog: http://blog.chrisrussell.net.

SCDL and Web Services

My Dad has been having fun hacking with single-board computers and has been sending me a lot of links. This review of the of the Raspberry Pi Model A looks interesting: A quick Model A show and tell from Adafruit

I spent years working as an embedded systems engineer and have a love of these small and inexpensive single-board computers. As a side note my Leap Motion Leap should arrive soon and I want to try connecting it to a Raspberry Pi for some recreational hacking.

The general idea is that I would use a small HTTP server on the Raspberry Pi to communicate with a fancy-pants HTML5 application running in the browser to monitor the Leap and control routing of Leap data out to various other peripheral devices connected to the Raspberry Pi. The details aren’t important. But what is interesting is how I think I can leverage SCDL in this project.

Soft Circuit Description Language  (SCDL) is a JSON object model for “soft circuits.” Soft circuits are system design artifacts conceptually analogous to digital circuit schematics.

As SCDL is JSON you can send it around conveniently. Maybe you have a web service API defined on your embedded web server that accepts a SCDL JSON parameter as input for example. Now consider what our little embedded web service might do with the SCDL it just received.

Recalling that SCDL is used to model program specifications, one particularly interesting thing our little embedded web server might do is transform the program specification into something it can actually execute, execute it, and return the results of the program execution to the caller as the request response.

Let that sink in for a minute. We just defined a single trivial web service API that can be implemented with widely available technology that effectively eliminates application logic as we know from the server (client too but we’ll get to that).

Rather than server-side routing tables and lots of carefully written routines here and there, we have a single simple web service that will essentially re-wire itself dynamically to service requests.

We can apply the same logic to the client as well and application logic as we know it starts to magically evaporate. Yet the functionality remains. There’s actually no magic. Just some clever factoring. But it’s weird. So let’s dig a bit deeper.

Think of a JSON-encoded SCDL message sent from point A to point B as a request and response protocol.

Request input comprises: a SCDL catalogue in JSON format, optionally some other JSON-encoded data objects that will be used as input to the program described by the SCDL catalogue.

Request output comprises: a JSON string (possibly containing another SCDL catalogue to be processed by the caller).

Bringing this back to the Raspberry Pi and our little embedded web server example, consider that upon receipt of a SCDL catalogue from the client, the embedded webserver unpacks the SCDL and uses the “soft circuit” schematic diagram it represents to:

  • instantiate the described program using a library of native plug-ins installed on the Raspberry Pi
  • “execute” this program (effectively a dynamically created web service)
  • return the results to the caller in JSON format (data + possibly more SCDL…)

Here the simple web service has absolutely no idea what the semantics of the request are.

And, this is why I say that there’s no hard-wired application logic on the server. In fact there is application logic on the server. It’s just expressed in a very pure and compact.

Our server-side deployment consists of (a) a collection of  low-level plug-ins and (b) a SCDL catalogue that specifies the “parts” that a client can use in a “soft circuit” to form a request.

In essence, this static SCDL catalogue is your application logic. It defines a grammar of possible requests and maps each to a concrete web service, built on-the-fly, via program transformation.

I like this idea because I’m lazy. I don’t want to figure out how to talk to my Leap device when it’s plugged into the USB port on my Raspberry Pi.

Rather, I want to browse a catalogue of Leap modules, select one for Raspberry Pi, stick it in my deployment package, and then expose it via my server’s SCDL catalogue. That way if a client wants to open a web socket to read a live telemetry stream of Leap data, it can send the server a SCDL request containing a little “soft circuit” that connects the Leap module to a web socket module.

Client-side, I’ve got the server’s SCDL catalog. I see the server has a Leap module and a web socket module and I know that the server knows how to connect the two somehow. I also know how I can ask the server to connect the these two resources for me.

We’ll come back to this topic later. That’s enough to think about for today :)

SCDL Visualization and Editing Ideas

Soft Circuit Description Language (SCDL) draws its core concepts from digital circuit schematics and defines a schema for serializing the following entities to JSON format:

  • virtual software “cores” – small, re-usable Turing Machine models
  • virtual software “chips” – re-usable system specification assemblies comprising any number of “cores” and/or “chips” wired together.
  • virtual software “circuits” – an abstract blueprint for building a software program.

The HTML5 SCDL editor I’m working on leverages D3js for data visualization (and some interaction). I am hugely inspired by this library.

Some ideas:

 

Relational input & graph building:

SCDL model hierarchy navigation:

SCDL interconnection topology:

SCDL entity interaction:

SCDL core transition matrix:

Simulation and test results: